Mobrovac has uploaded a new change for review.
https://gerrit.wikimedia.org/r/286246
Change subject: bin/ensure-kafka-topics-exist: Add DC awereness
......................................................................
bin/ensure-kafka-topics-exist: Add DC awereness
Messages in EventBus are being delivered by the proxy service to
DC-prefixed topics. This commit adds the functionality to create such
topics to the script.
Bug: T134015
Change-Id: Ifc3805c0f0bbc93e1741a8acc8d0720af8e5db50
---
M bin/ensure-kafka-topics-exist
1 file changed, 25 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/eventlogging
refs/changes/46/286246/1
diff --git a/bin/ensure-kafka-topics-exist b/bin/ensure-kafka-topics-exist
index 942468f..e186825 100755
--- a/bin/ensure-kafka-topics-exist
+++ b/bin/ensure-kafka-topics-exist
@@ -15,7 +15,7 @@
ap.add_argument(
'--topic-config',
- default='./config/topics.yaml',
+ default='./config/schemas/config/eventbus-topics.yaml',
help='Topic -> schema config file',
)
@@ -25,8 +25,21 @@
help='Comma-separated list of kafka broker hosts',
)
+ap.add_argument(
+ '--dc',
+ default='',
+ help='Comma-separated list of DC names to use as topic prefixes',
+)
-def ensure_topics_exist(kafka_client, topics):
+ap.add_argument(
+ '--no-dc',
+ action='store_true',
+ help='Don\'t use DC prefixes',
+)
+
+
+def ensure_topics_exist(kafka_client, topics, prefixes):
+ topics += [dc_prefix + '.' + topic for topic in topics for dc_prefix in
prefixes]
for topic in topics:
# Accessing the topic here will create it if
# it doesn't already exist.
@@ -60,5 +73,14 @@
with open(args.topic_config) as f:
topic_config = yaml.load(f)
+ # check the DC prefixes
+ if not args.dc and not args.no_dc:
+ logging.error('You must specify the DC names or use --no-dc!')
+ sys.exit(1)
+
kafka_client = KafkaClient(hosts=args.brokers)
- ensure_topics_exist(kafka_client, topic_config.keys())
+ if args.no_dc:
+ args.dc = []
+ else:
+ args.dc = args.dc.split(',')
+ ensure_topics_exist(kafka_client, topic_config.keys(), args.dc)
--
To view, visit https://gerrit.wikimedia.org/r/286246
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc3805c0f0bbc93e1741a8acc8d0720af8e5db50
Gerrit-PatchSet: 1
Gerrit-Project: eventlogging
Gerrit-Branch: master
Gerrit-Owner: Mobrovac <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
