Hey David, You might want to look at Aurora and Marathon to see how they do state reconciliation.
We are working on a new feature, adding persistent state to master (MESOS-764) <https://issues.apache.org/jira/browse/MESOS-764>, that should make reconciliation even easier.
