I think I found the problem.
There was a lone replica running in another DC. It was installed as a
replica some time ago with all the others. Think of this -- the
original config had 5 servers, one of them was this server. Then the
other 4 servers were RE-BUILT from scratch, so all the replication
agreements were changed AND - this is the important part - the 5th
server was never added back in. BUT - the 5th server was left running
and never told it that it was not a member anymore. It still thought it
had a replication agreement with original "server 1", but server 1 knew
Now, although the first 4 servers were rebuilt, the same domain, realm,
AND passwords were used.
I am guessing that somehow, this 5th server keeps trying to interject
its info into the ring of 4 servers, kind of forcing its way in.
Somehow, because the original credentials still work (but certs are all
different) is leaving the first 4 servers with a "can't decode" issue.
There should be some security checks so this can't happen. It should
also be easy to replicate.
Now I have to go re-initialize all the servers from a good server, so
everyone is happy again. The "problem" server has been shutdown
completely. (and yes, there were actually 3 of them in my scenario - I
just used 1 to simplify my example - but that explains the 3 CSNs that
just kept "appearing")
What concerns me most about this - were the servers outside of the "good
ring" somehow able to inject data into replication which might have been
causing bad data??? This is bad if it is true.
Manage your subscription for the Freeipa-users mailing list:
Go to http://freeipa.org for more info on the project