Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/252657
Change subject: statsv: migrate to pykafka
......................................................................
statsv: migrate to pykafka
python-kafka gets stuck; ottomata recommended pykafka as being less
error-prone.
Change-Id: I23c8c4962aa455bf885c7169319a9f9f0e4a5af9
---
M statsv.py
1 file changed, 14 insertions(+), 11 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/statsv
refs/changes/57/252657/1
diff --git a/statsv.py b/statsv.py
index e98a5f0..a95097c 100644
--- a/statsv.py
+++ b/statsv.py
@@ -30,21 +30,22 @@
import socket
import urlparse
-from kafka import KafkaClient, SimpleConsumer
+from pykafka import KafkaClient
logging.basicConfig(stream=sys.stderr, level=logging.INFO,
format='%(asctime)s %(message)s')
supported_metric_types = ('c', 'g', 'ms')
statsd_addr = ('statsd.eqiad.wmnet', 8125)
-kafka = KafkaClient((
- 'kafka1012.eqiad.wmnet',
- 'kafka1013.eqiad.wmnet',
- 'kafka1014.eqiad.wmnet',
- 'kafka1018.eqiad.wmnet',
- 'kafka1020.eqiad.wmnet',
- 'kafka1022.eqiad.wmnet',
-))
+
+kafka = KafkaClient(','.join((
+ 'kafka1012.eqiad.wmnet:9092',
+ 'kafka1013.eqiad.wmnet:9092',
+ 'kafka1014.eqiad.wmnet:9092',
+ 'kafka1018.eqiad.wmnet:9092',
+ 'kafka1020.eqiad.wmnet:9092',
+ 'kafka1022.eqiad.wmnet:9092',
+)))
def worker(q):
@@ -77,6 +78,8 @@
for _ in range(worker_count):
multiprocessing.Process(target=worker, args=(queue,)).start()
-consumer = SimpleConsumer(kafka, 'statsv', 'statsv')
+topic = kafka.topics['statsv']
+consumer = topic.get_simple_consumer()
for message in consumer:
- queue.put(message.message.value)
+ if message is not None:
+ queue.put(message.value)
--
To view, visit https://gerrit.wikimedia.org/r/252657
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I23c8c4962aa455bf885c7169319a9f9f0e4a5af9
Gerrit-PatchSet: 1
Gerrit-Project: analytics/statsv
Gerrit-Branch: master
Gerrit-Owner: Ori.livneh <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
