• 1: Select an ipa server to be authoritative and write the
contents of its database to an ldif file
On the master supplier:
/var/lib/dirsrv/scripts-EXAMPLE-COM/db2ldif.pl -D
'cn=Directory Manager' -w - -n userRoot -a
/tmp/master-389.ldif
Note that without the -r option it is deliberately ommiting
the tainted replication data which contains the bad CSNs
• 2: On the ipa server, shutdown its dirsrv daemon down so
that you can reset the attribute responsible for the serial
generation, and so that you can re-initialize its db from the
known good ldif
On the master supplier:
ipactl stop
• 3: Sanitize the dse.ldif Configuration File
On the master supplier:
edit the /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif file and
remove the nsState attribute from the replica config entry
You DO NOT want to remove the nsState from: dn: cn=uniqueid
generator,cn=config
The stanza you want to remove the value from is: dn:
cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config
The attribute will look like this: nsState::
cwAAAAAAAAA3QPBSAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==
Delete the entire line
• 3.1: Remove traces of stale CSN tracking in the Replica
Agreements themeselves
File location: /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
cat dse.ldif | sed -n '1 {h; $ !d}; $ {x; s/\n //g; p}; /^
/ {H; d}; /^ /! {x; s/\n //g; p}' | grep -v nsds50ruv >
new.dse.ldif
backup the old dse.ldif and replace it with the new one:
# mv dse.ldif dse.saved.ldif
# mv new.dse.ldif dse.ldif
• 4: Import the data from the known good ldif. This will mark
all the changes with CSNs that match the current time/date
stamps
On the master supplier:
chmod 644 /tmp/master-389.ldif
/var/lib/dirsrv/scripts-EXAMPLE-COM/ldif2db -n userRoot -i
/tmp/master-389.ldif
• 5: Restart the ipa daemons on the master supplier
#ipactl start
• 6: When the daemon starts, it will see that it does not have
an nsState and will write new CSN's to -all- of the newly
imported good data with today's timetamp, we need to take that
data and write -it- out to an ldif file
On the master supplier:
/var/lib/dirsrv/scripts-EXAMPLE-COM/db2ldif.pl -D
'cn=Directory Manager' -w - -n userRoot -r -a
/tmp/replication-master-389.ldif
^ the -r tells it to include all replica data which includes
the newly blessed CSN data
transfer the file to all of the ipa servers in the fleet
• 7: Now we must re-initialize _every other_ ipa consumer
server in the fleet with the new good data.
Steps 7-10 need to be done 1 at a time on each ipa consumer
server
ipactl stop
• 8: Sanitize the dse.ldif Configuration File
On the ipa server:
edit the /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif file and
remove the nsState attribute from the replica config entry
You DO NOT want to remove the nsState from: dn: cn=uniqueid
generator,cn=config
The stanza you want to remove the value from is: dn:
cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config
The attribute will look like this: nsState::
cwAAAAAAAAA3QPBSAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==
Delete the entire line
• 8.1: Remove traces of stale CSN tracking in the Replica
Agreements themeselves
File location: /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
cat dse.ldif | sed -n '1 {h; $ !d}; $ {x; s/\n //g; p}; /^
/ {H; d}; /^ /! {x; s/\n //g; p}' | grep -v nsds50ruv >
new.dse.ldif
backup the old dse.ldif and replace it with the new one
# mv dse.ldif dse.saved.ldif
# mv new.dse.ldif dse.ldif
• 9: Import the data from the known good ldif. This will mark
all the changes with CSNs that match the current time/date
stamps
On the auth server:
chmod 644 /tmp/replication-master-389.ldif
/var/lib/dirsrv/scripts-EXAMPLE-COM/ldif2db -n userRoot -i
/tmp/replication-master-389.ldif
• 10: Restart the ipa daemons on the ipa server
On the ipa server:
ipactl start
--------------------------------
From Rich Megginson:
Further reading for those interested in the particulars
of CSN tracking or the MultiMaster Replication algorithm,
you can read up about it here:
It all starts with the Leslie Lamport paper:
http://www.stanford.edu/class/cs240/readings/lamport.pdf
"Time, Clocks, and the Ordering of Events in a Distributed
System"
The next big impact on MMR protocols was the work done at
Xerox PARC on the Bayou project.
These and other sources formed the basis of the IETF LDUP
working group. Much of the MMR protocol is based on the
LDUP work.
The tl;dr version is this:
The MMR protocol is based on ordering operations by time
so that when you have two updates to the same attribute,
the "last one wins"
So how do you guarantee some sort of consistent ordering
throughout many systems that do not have clocks in sync
down to the millisecond? If you say "ntp" then you lose...
The protocol itself has to have some notion of time
differences among servers
The ordering is done by CSN (Change Sequence Number)
The first part of the CSN is the timestamp of the
operation in unix time_t (number of seconds since the
epoch).
In order to guarantee ordering, the MMR protocol has a
major constraint
You must never, never, issue a CSN that is the same or
less than another CSN
In order to guarantee that, the MMR protocol keeps track
of the time differences among _all_ of the servers that it
knows about.
When it generates CSNs, it uses the largest time
difference among all servers that it knows about.
So how does the time skew grow at all?
Due to timing differences, network latency, etc. the
directory server cannot always generate the absolute exact
system time. There will always be 1 or 2 second
differences in some replication sessions.
These 1 to 2 second differences accumulate over time.
However, there are things which can introduce really large
differences
1) buggy ntp implementations
2) bad sysadmin screws up the system clock
3) vms which are notorious for having laggy system clocks,
etc.
How can you monitor for this in the future?
The readnsState.py script supplied in this email can be
used to output the effective skew of the system date vs
the CSN generator.
You can set a crontab to run this script and monitor its
output to catch any future severe drifts.
Ticket information for some of the fixes that have been
implimented because of this work so far:
https://fedorahosted.org/389/ticket/47516
"You cannot hope to
secure that which you do not first understand"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
JR Aquino
Senior Information Security Specialist, Technical
Operations
T: +1 805 690 3478 | F: +1 805 879 3730 | M: +1 805 717
0365
GIAC Certified Exploit Researcher and Advanced Penetration
Tester |
GIAC WebApplication Penetration Tester | GIAC Certified
Incident Handler
[email protected]
Powering mobile workstyles and cloud services
