ryannedolan commented on pull request #9589: URL: https://github.com/apache/kafka/pull/9589#issuecomment-731307334
> Looking into my configuration, I now understand that I was lucky because I have one link going up back up replica_OLS->replica_CENTRAL which is now the single emitter of beats (which are then replicated in every other cluster) Ah, that's lucky indeed. If you sever that link, I suppose your heartbeats will stop flowing across the entire topology. Also, if you look at the heartbeat message content, you'll probably find that all heartbeats are originating from that same cluster. Sounds like disabling heartbeats altogether isn't what you want either. So we may need a way to emit heartbeats to source clusters without spinning up N*(N-1) herders just to do so. What if we created N additional herders just for heartbeats to each cluster? Say you had 20 clusters and only one flow enabled (A->B). In that case you could have 21 herders total: one for the A->B flow and 20 for per-cluster heartbeats. That's at least better than 20*19=380 herders. This raises the question of what "source" to associated with heartbeats, but seems like a solvable problem. Another idea is to have herders emit heartbeats _upstream_ within MirrorSourceConnector. In that case, the A->B herder would target B but, paradoxically, emit heartbeats to A. Hard to say if that would break anything, but as @twobeeb points out, this may be a reasonable interpretation of the documentation. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
