sascha-coenen commented on issue #8841: add readiness endpoints to processes 
having initialization delays
   BTW: readiness endpoints alone will not suffice to integrate Druid well with 
   Reason being that Kubernetes will only allow incoming requests to hit a 
service (e.g. a Historical ) after its readiness check passed. However, Druid 
uses zookeeper to discover if a new historical has joined the cluster. So as 
soon an a historical reports its existence to zookeeper (or to the coordinator 
if http announcement is configured), then the brokers think that the historical 
is ready for use and will start sending queries to it. However, those queries 
will not arrive at those historicals unless their readiness healthcheck has 
also passed already. So there is this intermittent timespan during which a 
broker cannot reach a historical and during this time all queries will fail 
because the broker will keep contacting the same historical, even if 
query-retries are configured. On retry, the same historical gets picked.
   I was wandering how to improve this behaviour during rolling updates.
   * One improvement would be to have the broker graylist a failing historical 
during retries.
   * Another improvement would be to add a configurable delay 
"druid.server.http.announcePropagationDelay" such that after a zookeeper 
announcement of a newly joined service (e.g. historical), there would be some 
delay until other services (e.g. brokers) would start to interact with the 
joined service. This would be similar to the existing property 
   The purpose of this would be that if Kubernetes is set up to execute a 
readiness check every 30 seconds, one could set the announcePropagationDelay to 
a larger time like 45 seconds, so that there would be no situation in which one 
Druid service tries to talk to another one at a time when Kubernetes is not 
(yet) allowing incoming traffic to the latter service.
   I just meant to mention it in this place because when I saw the title of 
this PR, especially the "having initialization delays" I hoped that the PR 
would also be about adding this internal delay.
   I have so far not found another workaround in kubernetes yet. It is 
certainly foremost a weakness in kubernetes that causes this misbehaviour 
because the readiness concept violates the single-responsibilty principle. A 
readiness check is used for two things:
   a) to decide whether incoming network connections are allowed for a given pod
   b) to decide whether a rolling update can already continue updating the next 
   Unfortunately there seems to be no way to have separate timings for these 
two unrelated aspects in k8s. 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to