The branch, master has been updated via 964bc8d markdown: Rename ms_markdown.py -> ms_schema_markdown.py via 4f20416 provision: Use the official MS 2008R2 schema by default via 1daba6f schema: 2008R2 AD schema attributes and classes via 8019c76 schema: 2016 AD schema attributes and classes via 8519f98 provision: RODC revision level should be at 2 via 1978838 selftest: Add basic test for schema upgrade via 6bdbcb1 domain.py: Auto-patch the diffs for the adprep schemaupgrade via 5db10e0 domain.py: Add a base dir option for schema upgrades via c870c34 schema: Some 2012 objects were missing systemflags via c22d022 upgradeprovision: Change test to always use 2008 R2 schema via ea9cde92 domain.py: Add base-schema option to samba-tool provision via 1f60f5b schema: Add option of specifying the base schema for a provision via f4d9b79 selftest: Fix upgradeprovision test by importing new objects for schema 45 via d157f97 2008R2: Missing flags on optional features container for objectVersion 45 via ff98bf9 2008R2: Missing extended rights for objectVersion 45 via d67f706 schema: Re-work extended rights handling in provision (prep for 2012R2) via d6e0f43 provision: Make clarifying header an LDIF comment in extended-rights.ldif via e8b200f provision: Align displayName of Property Sets with MS-ADTS 3.1.1.2.3.3 via d44c811 provision: Fill in a nicer displayName for Extended Rights via b9f0fbd provision: Fill in validAccesses in extended-rights.ldif for Property Sets via 7657168 provision: Fill in validAccesses in extended-rights.ldif for Validated Writes via 9840ee7 provision: Fill in validAccesses in extended-rights.ldif for Control Access Rights via 593a845 provision: Align extended-rights.ldif with the adprep LDIF for 2012R2 via 6721052 provision: Reformat appliesTo in Extended Rights into LDIF via 7fad489 provision: Remove section numbers from extended rights, replace with dn via 7bc9c20 provision: Import extended rights schema from MS-ADTS v47.0 via 9327c5a domain.py: Add a schemaupgrade option to apply missing 2008R2 schema via f9059c7 domain.py: Make schemaupgrade option work regardless of config via 580e6ba domain.py: Add schema upgrade option to samba-tool via 2650e92 schema: Allow schemaUpdateNow to refresh schema during a transaction via d66cbca adprep: Add the LDF data needed to upgrade to 2012R2 schema via d9c6f47 objectclass: Ensure that backlinks are not replicated via 3257c7f ms_schema: Properly handle base64 encoded attributes via 0f6e52a schema: 2012 and 2012 R2 AD schema attributes and classes via ed6a3dd ms_schema: Allow for CN=X and DC=X replacements via f4286f3 typo: Change case to match DN via 07f094f flags.h: Introduce the 2016 function level constant via 4ea7aa9 ldb: Show the last successful DN when failing to parse LDIF from e8b801d WHATSNEW: document the removal of 'auth methods', 'map untrusted to domain' and 'profile acls'
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 964bc8d19aa695f6c5188ab1a941127a259c0bc8 Author: Garming Sam <garm...@catalyst.net.nz> Date: Thu Nov 23 17:06:53 2017 +1300 markdown: Rename ms_markdown.py -> ms_schema_markdown.py We also reduce the scope of the import so that python-markdown is only required if interacting with 2012 code. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> Autobuild-User(master): Andrew Bartlett <abart...@samba.org> Autobuild-Date(master): Thu Dec 14 12:34:04 CET 2017 on sn-devel-144 commit 4f20416b383f11d6f7d30616696a5fdf5b6057a3 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Nov 20 17:10:25 2017 +1300 provision: Use the official MS 2008R2 schema by default This fixes us to have the official adminDescription etc. While both schema were provided by Microsoft this is a better quality one, but still under the same licence. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 1daba6f25541dab6f3d888431d00eb61544382d8 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Nov 20 15:45:41 2017 +1300 schema: 2008R2 AD schema attributes and classes Obtained under the Open Protocols Specifications licence from https://www.microsoft.com/en-us/download/details.aspx?id=23782 These are more complete than the version we have had in the tree until now. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 8019c76b5681a1a86b410fdd6bf0a1447266cfb8 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Nov 20 15:18:41 2017 +1300 schema: 2016 AD schema attributes and classes Obtained under the Open Protocols Specifications licence from https://www.microsoft.com/en-us/download/details.aspx?id=23782 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 8519f98677dd28b8ed4091bf266652b870cff4a8 Author: Garming Sam <garm...@catalyst.net.nz> Date: Wed Aug 16 16:02:32 2017 +1200 provision: RODC revision level should be at 2 This number had been mistakenly updated alongside the standard forest updates revision. This version number appears to be independent of the other revision levels. Also add the change to a new .ldf file, which can be used to apply the schema change to an existing Samba 4.7 (or earlier) instance. Update the provision/upgrade test to do just this (otherwise it complains about differences between a new provision and an older Samba 4.0.0 instance). Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 197883838f8ecb027e9d0375ff5238aec1567a42 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Fri Oct 6 16:30:40 2017 +1300 selftest: Add basic test for schema upgrade This tests that we can provision using both the 2008 and 2012 schema, that we can upgrade a 2008 Samba instance to use the 2012 schema, and that when we do that the result (more or less) matches a straight 2012 provision. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 6bdbcb1d4c577b019f74f027e5a753583f34bbf8 Author: Garming Sam <garm...@catalyst.net.nz> Date: Wed Nov 1 11:53:29 2017 +1300 domain.py: Auto-patch the diffs for the adprep schemaupgrade This creates a temporary directory where the markdown is parsed and the diffs are then applied. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 5db10e066279e8b2c917cad16908c66d36cbfde3 Author: Garming Sam <garm...@catalyst.net.nz> Date: Wed Nov 1 10:48:36 2017 +1300 domain.py: Add a base dir option for schema upgrades Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit c870c34df7fe1d4391543e6701a1398dce42c7e5 Author: Garming Sam <garm...@catalyst.net.nz> Date: Wed Sep 27 14:51:25 2017 +1300 schema: Some 2012 objects were missing systemflags The adprep LDIF files were adding the systemFlags, but they weren't present in the 2012 schema files. This is not just a Microsoft documentation problem - the difference was present when doing a provision of a 2012 Windows server vs using Adprep.exe to upgrade an older Windows server. Samba might as well use the correct systemFlags right from the start. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit c22d022ceafebe0c30a3947154016e51baf2d5b3 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Thu Oct 5 10:01:27 2017 +1300 upgradeprovision: Change test to always use 2008 R2 schema This tool (and the corresponding test) is designed to migrate a Samba DC from a pre-4.0.0 release up to a more recent schema (i.e. Windows 2008R2). Going further than 2008R2 turns this test into a bit of a nightmare. We now have a better adprep/'samba-tool domain schemaupgrade' option for upgrading from 2008R2 to a more recent schema. It seems to make most sense to leave this tests just running against 2008R2 schema provisions and add new tests to migrate from 2008R2 to 2012R2. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit ea9cde92fb57d6b65581b0fb48b8f3f253cadc55 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Thu Oct 5 09:53:28 2017 +1300 domain.py: Add base-schema option to samba-tool provision Allow a different base-schema to be used when provisioning a new domain. This allows us to test the new 2012 schema without committing Samba to using it by default. If, in future, we change the default to use the 2012 schema, some existing Samba tests (like upgradeprovision) rely on the 2012 schema. So making the base-schema optional allows these tests to continue using the older schema. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 1f60f5b51a8f510461f218ee1a5fc2ebbc9ac625 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Wed Oct 4 12:30:59 2017 +1300 schema: Add option of specifying the base schema for a provision Add the ability to override the base schema files being used for the new provision, e.g. instead of using the default supported schema, the code can now potentially specify an older or newer schema to use. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit f4d9b797e22a4cade3752930483bfc7a5a955338 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Dec 12 15:26:35 2017 +1300 selftest: Fix upgradeprovision test by importing new objects for schema 45 The recent schema changes mean that the upgradeprovision test starts failing. This is because it's using an old 4.0.0 schema (that doesn't have these schema changes), but it's comparing it against a fresh provision (which does have the changes). We can avoid this failure by using the 'samba-tool domain schemaupgrade' to bring the old 4.0.0 schema in line with a fresh provision. Note that the 'upgradeprovision --full' test doesn't need this change as it seems to more aggressively copy over any schema differences with a fresh provision. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit d157f9752bb3590ce74634db96d5c36b84ce792e Author: Andrew Bartlett <abart...@samba.org> Date: Tue Dec 12 15:20:26 2017 +1300 2008R2: Missing flags on optional features container for objectVersion 45 To match Windows 2008R2, this should have the same flags as the recycle bin enabled feature. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Andrew Bartlett <abart...@samba.org> commit ff98bf96e9b24242893dc0fe9e1f2fa64d261d30 Author: Andrew Bartlett <abart...@samba.org> Date: Wed Dec 13 15:03:57 2017 +1300 2008R2: Missing extended rights for objectVersion 45 We appear to have been missing some extended rights from 2008R2. These were added in samba by the extended-rights.ldif On Windows this was in Sch45.ldf (triggered by adprep schema updates). We add these changes to adprep/samba-4.7-missing-for-schema-45.ldif, which can be used to apply the changes to an existing Samba instance. This is not extracted from the Sch45.ldf file provided by Microsoft but is instead extracted using ldapcmp against a Samba install running the new extended-rights.ldif. Finally, these schema changes mean that the upgradeprovision test starts failing. This is because it's using an old 4.0.0 schema (that doesn't have these schema changes), but it's comparing it against a fresh provision (which does have the changes). We can avoid this failure by using the 'samba-tool domain schemaupgrade' to bring the old 4.0.0 schema in line with a fresh provision. Note that the 'upgradeprovision --full' test doesn't need this change as it seems to more aggressively copy over any schema differences with a fresh provision. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Andrew Bartlett <abart...@samba.org> commit d67f706b34d3bae05c7155092aa29d7e1148e7e6 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 14:42:55 2017 +1300 schema: Re-work extended rights handling in provision (prep for 2012R2) Add the changes needed to provision a 2012 DC (mostly this just affects the Extended Rights objects) by moving to the new extended-rights.ldif The localizationDisplayId is not documented in MS-ATDS so these values are moved to provision_configuation_modify.ldif and applied after the display-specifiers.ldif We don't enable the 2012R2 mode yet. The ${INC2012} variable just gets replaced with '#' so the lines get commented out and not applied. This approach allows us to support provisioning both a 2008R2 DC or a 2012R2 DC (so that we can test we can upgrade a 2008 DC to 2012). Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit d6e0f43ab98fff300ffdc7a888bbc84f74e580f0 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 14:50:39 2017 +1300 provision: Make clarifying header an LDIF comment in extended-rights.ldif Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit e8b200fad365298e57ca4b8fa7451e06451e7b0c Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 13:35:25 2017 +1300 provision: Align displayName of Property Sets with MS-ADTS 3.1.1.2.3.3 This gives some better names than what the CN of the object was. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit d44c811a8ca92347f29855909e1effc0c2c6abbd Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 13:26:53 2017 +1300 provision: Fill in a nicer displayName for Extended Rights We replace all the hyphens with a space. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit b9f0fbdeaa8571f3a3f382fee609402b4dddcbd8 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 12:35:45 2017 +1300 provision: Fill in validAccesses in extended-rights.ldif for Property Sets A Property Right has the value of RIGHT_DS_READ_PROPERTY|RIGHT_DS_WRITE_PROPERTY which is 48 (0x30) per 5.1.3.2 Access Rights. The property Sets are listed in MS-ATDS 3.1.1.2.3.3 and can also be found by looking at the attributeSecurityGuid on the schema objects. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 7657168e753c2c9de03e4a06fc89a1ff962e2f14 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 12:26:04 2017 +1300 provision: Fill in validAccesses in extended-rights.ldif for Validated Writes MS-ATDS 5.1.3.2.2 Validated Writes specifies the value of RIGHT_DS_WRITE_PROPERTY_EXTENDED which is 8 (0x08) per 5.1.3.2 Access Rights. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 9840ee76fbb2e52b2ddb36c9342eb9a7faeacda6 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 12:22:05 2017 +1300 provision: Fill in validAccesses in extended-rights.ldif for Control Access Rights MS-ATDS 5.1.3.2.1 Control Access Rights specifies the value of RIGHT_DS_CONTROL_ACCESS which is 256 (0x100) per 5.1.3.2 Access Rights. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 593a8456a8f194b1e0dd4fe4a6524375af5dc696 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 11:57:35 2017 +1300 provision: Align extended-rights.ldif with the adprep LDIF for 2012R2 This removes the additional rights for 2016 and flags the 2012R2 changes to allow the same file to be used to produce a 2008R2 or 2012R2 domain Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 67210522160ebc429b4eabc5f4e36d2677e145bb Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 11:09:51 2017 +1300 provision: Reformat appliesTo in Extended Rights into LDIF We remove comments about Schema 45 and earlier as this is the base level that Samba supports. A future commit will move to a machine-parsable flag for the 2012 schema and remove the 2016 elements. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 7fad4896f61f833d29a49ca20766abfbbd7874b4 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 10:51:32 2017 +1300 provision: Remove section numbers from extended rights, replace with dn This makes this file more like LDIF so we can process it automatically as well as use it as a text document. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 7bc9c20037c908a2211d92ddb960325a45972969 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 10:09:55 2017 +1300 provision: Import extended rights schema from MS-ADTS v47.0 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> commit 9327c5a35e760619060cf87b00ce39e32d54c319 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Thu Oct 5 16:16:30 2017 +1300 domain.py: Add a schemaupgrade option to apply missing 2008R2 schema We've identified some cases where we've gotten our implementation of the 2008R2 schema wrong. We can fix these up for new provisions going forward, but it'd be nice to have some way of fixing up the schema on existing DCs. A lot of what we're missing is already documented in Microsoft's Sch45.ldf file: https://technet.microsoft.com/en-us/library/dd378890(v=ws.10).aspx Unfortunately we can't just apply the Sch45.ldf file using the existing 'samba-tool domain schema-upgrade' option because: - We have got some of the Sch45.ldf changes, just not all of them. - We already say the Samba schema objectVersion is 47 (2008R2), so there's no way to tell if the Samba instance does or doesn't have the missing changes (apart from querying each change). We may want to add this to dbcheck eventually, but the simplest implementation option for now is to extend the new schemaupgrade command to allow us to specify a particular .LDF file to apply. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit f9059c7c1b83935dcd4b3bb645c926979c26a207 Author: Tim Beale <timbe...@catalyst.net.nz> Date: Thu Oct 5 15:43:53 2017 +1300 domain.py: Make schemaupgrade option work regardless of config Currently the 'samba-tool domain schemaupgrade' command will only work if the Samba config has the non-default option 'dsdb:schema update allowed = yes'. The whole point of running this samba-tool option is to upgrade the schema, so it would seem to make sense to bypass the setting temporarily, in order to apply the schema updates successfully. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 580e6babaf93a9a88e993527f0731408a0f2d9bf Author: Tim Beale <timbe...@catalyst.net.nz> Date: Wed Oct 4 12:30:59 2017 +1300 domain.py: Add schema upgrade option to samba-tool Microsoft has published the Schema updates that its Adprep.exe tool applies when it upgrades a 2008R2 schema to 2012R2. This patch adds an option to samba-tool to go through these update files and apply each change one by one. Along the way we need to make a few changes to the LDIF operations, e.g. change 'ntdsschemaadd' to 'add' and so on. The bulk of the changes involve parsing the .ldif file and separating out each update into a separate operation. There are a couple of errors that we've chosen to ignore: - Trying to set isDefunct for an object we don't know about. - Trying to set a value for an attribute OID that we don't know about (we may need to fix this in future, but it'll require some help from Microsoft about what the OIDs actually are). To try to make life easier, I've added a ldif_schema_update helper class. This provides convenient access of the DN the change applies to and other such details (whether it's setting isDefunct, etc). Pair-programmed-with: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 2650e9258b88228544148f5254dee7958819f6eb Author: Garming Sam <garm...@catalyst.net.nz> Date: Fri Aug 18 13:59:30 2017 +1200 schema: Allow schemaUpdateNow to refresh schema during a transaction When we upgrade a schema from 2008R2 to 2012R2, we want to apply all the changes in a single transaction - if we can't apply all the updates then we don't want to be left with a schema halfway in between the two. However, as we apply each LDIF update, we also want to refresh the schema. There are 2 reasons for this: 1. The adprep .LDIF files provided by Microsoft have some writes to schemaUpdateNow in them. 2. Microsoft uses attribute OIDs in their adprep .LDIF files, which Samba doesn't handle so well. However, we can replace the OIDs with the attribute's ldapDisplayName and they work fine. But to do this, we need to query the schema to map the OID to attribute name. And to query the schema successfully, the schema needs to be refreshed after the new attribute object has been added. Basically this patch avoids bailing out during the dsdb_schema_refresh() if we are writing schemaUpdateNow as part of a larger transaction. Pair-programmed-with: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit d66cbca4e1c4d68a40fb117efc3e7e357690fc5d Author: Garming Sam <garm...@catalyst.net.nz> Date: Tue Oct 3 10:01:30 2017 +1300 adprep: Add the LDF data needed to upgrade to 2012R2 schema This patch adds the LDF files corresponding to the changes that the Windows Adprep.exe tool makes when upgrading a AD schema to Windows 2012R2. This is based on information Microsoft has made public on github (Schema-Updates.md - see the README.txt for more details). The LDF files 48-56 are for upgrading to Windows Server 2012, and 57-69 are for Windows Server 2012 R2. Unfortunately, the raw LDF information from Microsoft wasn't enough to get the schema working. The .diff files contain changes we needed to make on top of the raw LDF content from Microsoft. The basic steps to regenerate the .LDF files are documented in the README.txt file. The files used to generate the .LDF files are in the WindowsServerDocs/ sub-directory. (The .LDF generation is done at runtime during provision). Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit d9c6f47851311838ba6a67f8effc7a4e398e12ca Author: Garming Sam <garm...@catalyst.net.nz> Date: Tue Sep 5 16:03:04 2017 +1200 objectclass: Ensure that backlinks are not replicated Adprep schema adds backlinks, but they do not have the NOT_REPLICATED bit. We need to force this in locally to ensure we have it. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 3257c7f60fea45ade3f761dd566aa181417b638c Author: Garming Sam <garm...@catalyst.net.nz> Date: Fri Aug 18 13:46:57 2017 +1200 ms_schema: Properly handle base64 encoded attributes There used to be a special case for omobjectclass, but now there is just generic handling for such attributes. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 0f6e52a268c9812e12602031247c88cceb9dec62 Author: Garming Sam <garm...@catalyst.net.nz> Date: Mon Sep 12 17:07:02 2016 +1200 schema: 2012 and 2012 R2 AD schema attributes and classes Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit ed6a3ddb2ac127e086af748710750320d46f13d4 Author: Garming Sam <garm...@catalyst.net.nz> Date: Mon Sep 19 13:52:54 2016 +1200 ms_schema: Allow for CN=X and DC=X replacements These occur in the newer 2012 and 2016 schemas. Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit f4286f3516c3e14ebdd16758cadc4ed4c0afce10 Author: Garming Sam <garm...@catalyst.net.nz> Date: Wed Aug 2 12:52:22 2017 +1200 typo: Change case to match DN Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 07f094f69fa91f7f363ca892cd2a640a76c90a94 Author: Garming Sam <garm...@catalyst.net.nz> Date: Tue Aug 15 15:17:34 2017 +1200 flags.h: Introduce the 2016 function level constant Signed-off-by: Garming Sam <garm...@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 4ea7aa9265199e515d8f08ef849b69cfa3ee1955 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Dec 11 15:57:30 2017 +1300 ldb: Show the last successful DN when failing to parse LDIF Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Garming Sam <garm...@catalyst.net.nz> ----------------------------------------------------------------------- Summary of changes: lib/ldb/pyldb.c | 24 +- libds/common/flags.h | 1 + python/samba/ms_schema.py | 52 +- python/samba/ms_schema_markdown.py | 71 + python/samba/netcmd/domain.py | 309 +- python/samba/provision/__init__.py | 18 +- python/samba/schema.py | 41 +- python/samba/upgradehelpers.py | 4 +- selftest/tests.py | 5 + source4/dsdb/pydsdb.c | 1 + source4/dsdb/samdb/ldb_modules/objectclass.c | 6 + source4/dsdb/samdb/ldb_modules/rootdse.c | 16 + source4/dsdb/samdb/ldb_modules/schema_load.c | 12 +- source4/scripting/bin/samba_upgradeprovision | 2 +- .../AD_DS_Attributes__Windows_Server_2012_R2.ldf | 30374 ++++++++++++++++++ .../AD_DS_Attributes__Windows_Server_2016.ldf | 30912 +++++++++++++++++++ .../AD_DS_Classes__Windows_Server_2012_R2.ldf | 8875 ++++++ .../AD_DS_Classes__Windows_Server_2016.ldf | 9031 ++++++ ...ttributes_for_AD_DS__Windows_Server_2008_R2.ldf | 26925 ++++++++++++++++ .../Attributes_for_AD_DS__Windows_Server_2012.ldf | 29357 ++++++++++++++++++ .../Classes_for_AD_DS__Windows_Server_2008_R2.ldf | 7934 +++++ .../Classes_for_AD_DS__Windows_Server_2012.ldf | 8624 ++++++ source4/setup/adprep/README.txt | 23 + source4/setup/adprep/WindowsServerDocs/LICENSE | 395 + .../setup/adprep/WindowsServerDocs/LICENSE-CODE | 17 + .../setup/adprep/WindowsServerDocs/Sch49.ldf.diff | 30 + .../setup/adprep/WindowsServerDocs/Sch50.ldf.diff | 107 + .../setup/adprep/WindowsServerDocs/Sch51.ldf.diff | 225 + .../setup/adprep/WindowsServerDocs/Sch57.ldf.diff | 105 + .../setup/adprep/WindowsServerDocs/Sch59.ldf.diff | 26 + .../adprep/WindowsServerDocs/Schema-Updates.md | 4583 +++ source4/setup/adprep/fix-forest-rev.ldf | 6 + .../adprep/samba-4.7-missing-for-schema45.ldif | 112 + source4/setup/extended-rights.ldif | 835 + source4/setup/provision_configuration.ldif | 682 +- source4/setup/provision_configuration_modify.ldif | 503 + source4/setup/provision_schema_basedn_modify.ldif | 2 +- source4/setup/provision_users.ldif | 4 +- source4/setup/wscript_build | 3 + testprogs/blackbox/schemaupgrade.sh | 122 + testprogs/blackbox/upgradeprovision-oldrelease.sh | 8 +- 41 files changed, 159669 insertions(+), 713 deletions(-) create mode 100644 python/samba/ms_schema_markdown.py create mode 100644 source4/setup/ad-schema/AD_DS_Attributes__Windows_Server_2012_R2.ldf create mode 100644 source4/setup/ad-schema/AD_DS_Attributes__Windows_Server_2016.ldf create mode 100644 source4/setup/ad-schema/AD_DS_Classes__Windows_Server_2012_R2.ldf create mode 100644 source4/setup/ad-schema/AD_DS_Classes__Windows_Server_2016.ldf create mode 100644 source4/setup/ad-schema/Attributes_for_AD_DS__Windows_Server_2008_R2.ldf create mode 100644 source4/setup/ad-schema/Attributes_for_AD_DS__Windows_Server_2012.ldf create mode 100644 source4/setup/ad-schema/Classes_for_AD_DS__Windows_Server_2008_R2.ldf create mode 100644 source4/setup/ad-schema/Classes_for_AD_DS__Windows_Server_2012.ldf create mode 100644 source4/setup/adprep/README.txt create mode 100644 source4/setup/adprep/WindowsServerDocs/LICENSE create mode 100644 source4/setup/adprep/WindowsServerDocs/LICENSE-CODE create mode 100644 source4/setup/adprep/WindowsServerDocs/Sch49.ldf.diff create mode 100644 source4/setup/adprep/WindowsServerDocs/Sch50.ldf.diff create mode 100644 source4/setup/adprep/WindowsServerDocs/Sch51.ldf.diff create mode 100644 source4/setup/adprep/WindowsServerDocs/Sch57.ldf.diff create mode 100644 source4/setup/adprep/WindowsServerDocs/Sch59.ldf.diff create mode 100644 source4/setup/adprep/WindowsServerDocs/Schema-Updates.md create mode 100644 source4/setup/adprep/fix-forest-rev.ldf create mode 100644 source4/setup/adprep/samba-4.7-missing-for-schema45.ldif create mode 100644 source4/setup/extended-rights.ldif create mode 100755 testprogs/blackbox/schemaupgrade.sh Changeset truncated at 500 lines: diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c index e61b5b6..04b3f1b 100644 --- a/lib/ldb/pyldb.c +++ b/lib/ldb/pyldb.c @@ -1669,6 +1669,7 @@ static PyObject *py_ldb_parse_ldif(PyLdbObject *self, PyObject *args) PyObject *list, *ret; struct ldb_ldif *ldif; const char *s; + struct ldb_dn *last_dn = NULL; TALLOC_CTX *mem_ctx; @@ -1686,8 +1687,29 @@ static PyObject *py_ldb_parse_ldif(PyLdbObject *self, PyObject *args) talloc_steal(mem_ctx, ldif); if (ldif) { PyList_Append(list, ldb_ldif_to_pyobject(ldif)); + last_dn = ldif->msg->dn; } else { - PyErr_SetString(PyExc_ValueError, "unable to parse ldif string"); + const char *last_dn_str = NULL; + const char *err_string = NULL; + if (last_dn == NULL) { + PyErr_SetString(PyExc_ValueError, + "unable to parse LDIF " + "string at first chunk"); + talloc_free(mem_ctx); + return NULL; + } + + last_dn_str + = ldb_dn_get_linearized(last_dn); + + err_string + = talloc_asprintf(mem_ctx, + "unable to parse ldif " + "string AFTER %s", + last_dn_str); + + PyErr_SetString(PyExc_ValueError, + err_string); talloc_free(mem_ctx); return NULL; } diff --git a/libds/common/flags.h b/libds/common/flags.h index 88b93cb..d431bd5 100644 --- a/libds/common/flags.h +++ b/libds/common/flags.h @@ -190,6 +190,7 @@ #define DS_DOMAIN_FUNCTION_2008_R2 4 #define DS_DOMAIN_FUNCTION_2012 5 #define DS_DOMAIN_FUNCTION_2012_R2 6 +#define DS_DOMAIN_FUNCTION_2016 7 /* sa->systemFlags on attributes */ #define DS_FLAG_ATTR_NOT_REPLICATED 0x00000001 diff --git a/python/samba/ms_schema.py b/python/samba/ms_schema.py index 245ce3f..a8c9363 100644 --- a/python/samba/ms_schema.py +++ b/python/samba/ms_schema.py @@ -162,6 +162,10 @@ def fix_dn(dn): dn = dn.replace("\n ", "") dn = dn.replace(" ", "") return dn.replace("CN=Schema,CN=Configuration,<RootDomainDN>", "${SCHEMADN}") + elif dn.endswith("DC=X"): + return dn.replace("CN=Schema,CN=Configuration,DC=X", "${SCHEMADN}") + elif dn.endswith("CN=X"): + return dn.replace("CN=Schema,CN=Configuration,CN=X", "${SCHEMADN}") else: return dn @@ -194,7 +198,7 @@ def __write_ldif_one(entry): else: vl = l[1] - if l[0].lower() == 'omobjectclass': + if l[2]: out.append("%s:: %s" % (l[0], l[1])) continue @@ -211,8 +215,15 @@ def __transform_entry(entry, objectClass): entry = [l.split(":", 1) for l in entry] cn = "" + skip_dn = skip_objectclass = skip_admin_description = skip_admin_display_name = False for l in entry: + if l[1].startswith(': '): + l.append(True) + l[1] = l[1][2:] + else: + l.append(False) + key = l[0].lower() l[1] = l[1].lstrip() l[1] = l[1].rstrip() @@ -231,25 +242,42 @@ def __transform_entry(entry, objectClass): l[1] = __convert_bitfield(key, l[1]) if key == "omobjectclass": - l[1] = oMObjectClassBER[l[1].strip()] + if not l[2]: + l[1] = oMObjectClassBER[l[1].strip()] + l[2] = True if isinstance(l[1], str): l[1] = fix_dn(l[1]) + if key == 'dn': + skip_dn = True + dn = l[1] + + if key == 'objectclass': + skip_objectclass = True + elif key == 'admindisplayname': + skip_admin_display_name = True + elif key == 'admindescription': + skip_admin_description = True assert(cn) - entry.insert(0, ["dn", "CN=%s,${SCHEMADN}" % cn]) - entry.insert(1, ["objectClass", ["top", objectClass]]) - entry.insert(2, ["cn", cn]) - entry.insert(2, ["objectGUID", str(uuid.uuid4())]) - entry.insert(2, ["adminDescription", cn]) - entry.insert(2, ["adminDisplayName", cn]) - for l in entry: - key = l[0].lower() + header = [] + if not skip_dn: + header.append(["dn", "CN=%s,${SCHEMADN}" % cn, False]) + else: + header.append(["dn", dn, False]) + + if not skip_objectclass: + header.append(["objectClass", ["top", objectClass], False]) + if not skip_admin_description: + header.append(["adminDescription", cn, False]) + if not skip_admin_display_name: + header.append(["adminDisplayName", cn, False]) + + header.append(["objectGUID", str(uuid.uuid4()), False]) - if key == "cn": - entry.remove(l) + entry = header + [x for x in entry if x[0].lower() not in {'dn', 'changetype', 'objectcategory'}] return entry diff --git a/python/samba/ms_schema_markdown.py b/python/samba/ms_schema_markdown.py new file mode 100644 index 0000000..c695f8b --- /dev/null +++ b/python/samba/ms_schema_markdown.py @@ -0,0 +1,71 @@ +# Create schema.ldif from Github markdown +# +# Each LDF section in the markdown file then gets written to a corresponding +# .LDF output file. +# +# Copyright (C) Andrew Bartlett 2017 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +"""Generate LDIF from Github documentation.""" + +import re +import os +import markdown +import xml.etree.ElementTree as ET + +def innertext(tag): + return (tag.text or '') + \ + ''.join(innertext(e) for e in tag) + \ + (tag.tail or '') + +def read_ms_markdown(in_file, out_folder): + """Read Github documentation-derived schema files.""" + + with open(in_file) as update_file: + # Remove any comments from the raw LDF files + html = markdown.markdown(re.sub(r'(?m)^# .*\n?', '', update_file.read()), + output_format='xhtml') + + tree = ET.fromstring('<root>' + html + '</root>') + + ldf = None + try: + for node in tree: + if node.tag == 'h3': + if ldf is not None: + ldf.close() + + out_path = os.path.join(out_folder, innertext(node).strip()) + ldf = open(out_path, 'w') + elif node.tag == 'p' and ldf is not None: + ldf.write(innertext(node).replace('```', '') + '\n') + finally: + if ldf is not None: + ldf.close() + +if __name__ == '__main__': + import sys + + out_folder = '' + + if len(sys.argv) == 0: + print >>sys.stderr, "Usage: %s <Schema-Update.md> [<output folder>]" % (sys.argv[0]) + sys.exit(1) + + in_file = sys.argv[1] + if len(sys.argv) > 2: + out_folder = sys.argv[2] + + read_ms_markdown(in_file, out_folder) diff --git a/python/samba/netcmd/domain.py b/python/samba/netcmd/domain.py index f54b404..6f6ef61 100644 --- a/python/samba/netcmd/domain.py +++ b/python/samba/netcmd/domain.py @@ -33,6 +33,7 @@ import tempfile import logging import subprocess import time +import shutil from samba import ntstatus from samba import NTSTATUSError from samba import werror @@ -85,7 +86,8 @@ from samba.dsdb import ( from samba.provision import ( provision, ProvisioningError, - DEFAULT_MIN_PWD_LENGTH + DEFAULT_MIN_PWD_LENGTH, + setup_path ) from samba.provision.common import ( @@ -232,6 +234,10 @@ class cmd_domain_provision(Command): choices=["2000", "2003", "2008", "2008_R2"], help="The domain and forest function level (2000 | 2003 | 2008 | 2008_R2 - always native). Default is (Windows) 2008_R2 Native.", default="2008_R2"), + Option("--base-schema", type="choice", metavar="BASE-SCHEMA", + choices=["2008_R2", "2008_R2_old", "2012", "2012_R2"], + help="The base schema files to use. Default is (Windows) 2008_R2.", + default="2008_R2"), Option("--next-rid", type="int", metavar="NEXTRID", default=1000, help="The initial nextRid value (only needed for upgrades). Default is 1000."), Option("--partitions-only", @@ -309,7 +315,8 @@ class cmd_domain_provision(Command): ldap_backend_nosync=None, ldap_backend_extra_port=None, ldap_backend_forced_uri=None, - ldap_dryrun_mode=None): + ldap_dryrun_mode=None, + base_schema=None): self.logger = self.get_logger("provision") if quiet: @@ -477,7 +484,8 @@ class cmd_domain_provision(Command): use_rfc2307=use_rfc2307, skip_sysvolacl=False, ldap_backend_extra_port=ldap_backend_extra_port, ldap_backend_forced_uri=ldap_backend_forced_uri, - nosync=ldap_backend_nosync, ldap_dryrun_mode=ldap_dryrun_mode) + nosync=ldap_backend_nosync, ldap_dryrun_mode=ldap_dryrun_mode, + base_schema=base_schema) except ProvisioningError, e: raise CommandError("Provision failed", e) @@ -3852,6 +3860,300 @@ class cmd_domain_tombstones(SuperCommand): subcommands = {} subcommands["expunge"] = cmd_domain_tombstones_expunge() +class ldif_schema_update: + """Helper class for applying LDIF schema updates""" + + def __init__(self): + self.is_defunct = False + self.unknown_oid = None + self.dn = None + self.ldif = "" + + def _ldap_schemaUpdateNow(self, samdb): + ldif = """ +dn: +changetype: modify +add: schemaUpdateNow +schemaUpdateNow: 1 +""" + samdb.modify_ldif(ldif) + + def can_ignore_failure(self, error): + """Checks if we can safely ignore failure to apply an LDIF update""" + (num, errstr) = error.args + + # Microsoft has marked objects as defunct that Samba doesn't know about + if num == ldb.ERR_NO_SUCH_OBJECT and self.is_defunct: + print("Defunct object %s doesn't exist, skipping" % self.dn) + return True + elif self.unknown_oid is not None: + print("Skipping unknown OID %s for object %s" %(self.unknown_oid, self.dn)) + return True + + return False + + def apply(self, samdb): + """Applies a single LDIF update to the schema""" + + try: + samdb.modify_ldif(self.ldif, controls=['relax:0']) + except ldb.LdbError as e: + if self.can_ignore_failure(e): + return 0 + else: + print("Exception: %s" % e) + print("Encountered while trying to apply the following LDIF") + print("----------------------------------------------------") + print("%s" % self.ldif) + + raise + + # REFRESH AFTER EVERY CHANGE + # Otherwise the OID-to-attribute mapping in _apply_updates_in_file() + # won't work, because it can't lookup the new OID in the schema + self._ldap_schemaUpdateNow(samdb) + + return 1 + +class cmd_domain_schema_upgrade(Command): + """Domain schema upgrading""" + + synopsis = "%prog [options]" + + takes_optiongroups = { + "sambaopts": options.SambaOptions, + "versionopts": options.VersionOptions, + "credopts": options.CredentialsOptions, + } + + takes_options = [ + Option("-H", "--URL", help="LDB URL for database or target server", type=str, + metavar="URL", dest="H"), + Option("--quiet", help="Be quiet", action="store_true"), + Option("--verbose", help="Be verbose", action="store_true"), + Option("--schema", type="choice", metavar="SCHEMA", + choices=["2012", "2012_R2"], + help="The schema file to upgrade to. Default is (Windows) 2012_R2.", + default="2012_R2"), + Option("--ldf-file", type=str, default=None, + help="Just apply the schema updates in the adprep/.LDF file(s) specified"), + Option("--base-dir", type=str, default=None, + help="Location of ldf files Default is ${SETUPDIR}/adprep.") + ] + + def _apply_updates_in_file(self, samdb, ldif_file): + """ + Applies a series of updates specified in an .LDIF file. The .LDIF file + is based on the adprep Schema updates provided by Microsoft. + """ + count = 0 + ldif_op = ldif_schema_update() + + # parse the file line by line and work out each update operation to apply + for line in ldif_file: + + line = line.rstrip() + + # the operations in the .LDIF file are separated by blank lines. If + # we hit a blank line, try to apply the update we've parsed so far + if line == '': + + # keep going if we haven't parsed anything yet + if ldif_op.ldif == '': + continue + + # Apply the individual change + count += ldif_op.apply(samdb) + + # start storing the next operation from scratch again + ldif_op = ldif_schema_update() + continue + + # replace the placeholder domain name in the .ldif file with the real domain + if line.upper().endswith('DC=X'): + line = line[:-len('DC=X')] + str(samdb.get_default_basedn()) + elif line.upper().endswith('CN=X'): + line = line[:-len('CN=X')] + str(samdb.get_default_basedn()) + + values = line.split(':') + + if values[0].lower() == 'dn': + ldif_op.dn = values[1].strip() + + # replace the Windows-specific operation with the Samba one + if values[0].lower() == 'changetype': + line = line.lower().replace(': ntdsschemaadd', + ': add') + line = line.lower().replace(': ntdsschemamodify', + ': modify') + + if values[0].lower() in ['rdnattid', 'subclassof', + 'systemposssuperiors', + 'systemmaycontain', + 'systemauxiliaryclass']: + _, value = values + + # The Microsoft updates contain some OIDs we don't recognize. + # Query the DB to see if we can work out the OID this update is + # referring to. If we find a match, then replace the OID with + # the ldapDisplayname + if '.' in value: + res = samdb.search(base=samdb.get_schema_basedn(), + expression="(|(attributeId=%s)(governsId=%s))" % + (value, value), + attrs=['ldapDisplayName']) + + if len(res) != 1: + ldif_op.unknown_oid = value + else: + display_name = res[0]['ldapDisplayName'][0] + line = line.replace(value, ' ' + display_name) + + # Microsoft has marked objects as defunct that Samba doesn't know about + if values[0].lower() == 'isdefunct' and values[1].strip().lower() == 'true': + ldif_op.is_defunct = True + + # Samba has added the showInAdvancedViewOnly attribute to all objects, + # so rather than doing an add, we need to do a replace + if values[0].lower() == 'add' and values[1].strip().lower() == 'showinadvancedviewonly': + line = 'replace: showInAdvancedViewOnly' + + # Add the line to the current LDIF operation (including the newline + # we stripped off at the start of the loop) + ldif_op.ldif += line + '\n' + + return count + + + def _apply_update(self, samdb, update_file, base_dir): + """Wrapper function for parsing an LDIF file and applying the updates""" + + print("Applying %s updates..." % update_file) + + ldif_file = None + try: + ldif_file = open(os.path.join(base_dir, update_file)) + + count = self._apply_updates_in_file(samdb, ldif_file) + + finally: + if ldif_file: + ldif_file.close() + + print("%u changes applied" % count) + + return count + + def run(self, **kwargs): + from samba.ms_schema_markdown import read_ms_markdown + from samba.schema import Schema + + updates_allowed_overriden = False + sambaopts = kwargs.get("sambaopts") + credopts = kwargs.get("credopts") + versionpts = kwargs.get("versionopts") + lp = sambaopts.get_loadparm() + creds = credopts.get_credentials(lp) + H = kwargs.get("H") + target_schema = kwargs.get("schema") + ldf_files = kwargs.get("ldf_file") + base_dir = kwargs.get("base_dir") + + temp_folder = None + + samdb = SamDB(url=H, session_info=system_session(), credentials=creds, lp=lp) + + # we're not going to get far if the config doesn't allow schema updates + if lp.get("dsdb:schema update allowed") is None: + lp.set("dsdb:schema update allowed", "yes") + print("Temporarily overriding 'dsdb:schema update allowed' setting") + updates_allowed_overriden = True + + # if specific LDIF files were specified, just apply them + if ldf_files: + schema_updates = ldf_files.split(",") + else: + schema_updates = [] + + # work out the version of the target schema we're upgrading to + end = Schema.get_version(target_schema) -- Samba Shared Repository