michael-trelinski commented on a change in pull request #6740: Zookeeper loss
URL: https://github.com/apache/incubator-druid/pull/6740#discussion_r257453230
##########
File path: server/src/main/java/org/apache/druid/curator/CuratorModule.java
##########
@@ -79,10 +82,33 @@ public CuratorFramework makeCurator(CuratorConfig config,
EnsembleProvider ensem
StringUtils.format("%s:%s", config.getZkUser(),
config.getZkPwd()).getBytes(StandardCharsets.UTF_8)
);
}
+
+ RetryPolicy retryPolicy;
+ if (config.getTerminateDruidProcessOnConnectFail()) {
+ final Runnable exitRunner = () -> {
+ try {
+ log.error("Zookeeper can't be reached, forcefully stopping
lifecycle...");
+ lifecycle.stop();
+ System.err.println("Zookeeper can't be reached, forcefully stopping
virtual machine...");
+ }
+ finally {
+ System.exit(1);
Review comment:
> Correct me if I'm wrong, please, we could have the whole Druid clusters
restarted due to a ZK cluster shortage?
@egor-ryashin That's the point. It is **disabled by default**. A lot of us
want to have the Java process die (so that an outside process can restart it,
like other popular Linux daemons). Some scenarios cause it to not completely
die, because a Zookeeper thread hangs on forever.
Our use case was this: DNS/IT services changed the IPs of the zookeeper
cluster after the JVM had already cached them. Had they been able to die
gracefully, a daemon would have caused the cluster to "self heal." This use
case is not an exhaustive list.
Also, we noticed that queries gave bad data during these times. Sometimes,
for some use cases (as is ours), no data is preferable to bad data.
> For me it looks like a hack, shouldn't we try to recreate a
CuratorFramework?
This would not have fixed the issue in our use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
