I'd love to get some insights on how things work at linkedin in terms of your web servers and kafka producers.
You guys probably connect to multiple kafka clusters, so let's assume you are only connecting to a single cluster. 1. do you use a single producer for all message types/topics? 2. For your pageview topic i.e. it is getting sent on a per page request (albiet it is batched): *What happens when your kafka cluster is down? Will your web application behave as normal or will it really slow things down? Locally on my laptop I shutdown my vagrant that is running kafka, and the page renders very slow when the producer is down. Or do you use some smart circuit breaker logic that will stop trying to send producer messages if kafka is down?
