Gehel has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286397
Change subject: Make es-tool more robust when checking for cluster health
......................................................................
Make es-tool more robust when checking for cluster health
A number of errors can occur when checking for clsuter health. Timeout when
connecting to elasticsearch, master not elected, ... All those error condition
should be treated as cluster not healthy.
Change-Id: I39d553100bff2b1f6d6f2c517b6ce398300dbe37
---
M modules/elasticsearch/files/es-tool
1 file changed, 22 insertions(+), 6 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/97/286397/1
diff --git a/modules/elasticsearch/files/es-tool
b/modules/elasticsearch/files/es-tool
index 7b21918..58276f7 100755
--- a/modules/elasticsearch/files/es-tool
+++ b/modules/elasticsearch/files/es-tool
@@ -242,16 +242,31 @@
# Wait a bit
time.sleep(5)
+ es_wait_for_green()
+
+ return os.EX_OK
+
+
+def es_wait_for_green():
print "Waiting for green (you can ctrl+c here if you have to)...\n"
- while cluster_health() != "green":
- print '\n'.join(cluster_status(columns=('status',
- 'initializing_shards',
- 'relocating_shards',
- 'unassigned_shards')))
+ while not is_cluster_healthy():
+ try:
+ print '\n'.join(cluster_status(columns=('status',
+ 'initializing_shards',
+ 'relocating_shards',
+ 'unassigned_shards')))
+ except:
+ printu("Cannot print cluster status\n")
time.sleep(60)
print "ok"
- return os.EX_OK
+
+def is_cluster_healthy():
+ try:
+ return cluster_health() == "green"
+ except:
+ printu("Error while checking for cluster health\n")
+ return False
def es_upgrade_fast():
@@ -333,6 +348,7 @@
"start-replication": es_start_replication,
"stop-replication": es_stop_replication,
"unban-node": es_unban_node,
+ "wait-for-green": es_wait_for_green,
"status": lambda: '\n'.join(cluster_status()),
}
--
To view, visit https://gerrit.wikimedia.org/r/286397
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I39d553100bff2b1f6d6f2c517b6ce398300dbe37
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gehel <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
