As discussed w/ rnewson in #couchdb I accidentally used the same [couchdb] uuid on multiple servers. This never made it to my production system because I noticed errors like "Error updating the target checkpoint document: conflict" in the log, and then noticed that replication_id was not unique per instance when looking at http request logs. I ended up specifying private-ip:port in my script that triggers replication which leads to the replication_id being unique for each instance on its peers.
The explanation for how I ended up using the same [couchdb] uuid on multiple servers is that I use puppet to deploy couchdb servers, and I manage a copy of local.ini in puppet. I built couchdb once withouth puppet to generate the local.ini / default.ini, and then I took copies of these and put them in puppet. I overlooked that [couchdb] uuid was defined in the copy of local.ini. This copy of local.ini with uuid defined in it was then getting pushed out to my testing machines, causing them to all use the same replication_id on their peers. rnewson asked me to write to the list about the above, in order to discuss storing uuid in a place where it may be less likely to be copied in deployment management systems / scripts like Puppet.
