I have questions regarding spark structured streaming deployment

I have +- 100 kafka topics that can be processed using similar code block.
I am using pyspark 2.2.1

Here is the idea:

TOPIC_LIST = ["topic_a","topic_b"."topic_c"]

stream = {}
for t in TOPIC_LIST:
    stream[t] = spark \
                .readStream \
                .format("kafka") \
"kafka-broker1:9092,kafka-broker2:9092") \
                .option("subscribe", "{0}".format(t)) \
                .load().writeStream \
                .trigger(processingTime="10 seconds") \
                .format("console") \

So far, I have these options:
1. do a single spark submit for all topics
2. do multiple spark submit per topics

I am thinking of deploying this using supervisor or upstart so when spark
app is down, I can get notified.

By submitting single spark app I can have more control on server resources
but a little bit hard to monitor if one app is having problem. For multiple
spark submits I am afraid I need to provide +- 100 cpu cores which I doubt
will be fully utilized as some of the topics do not have high througput,
however by creating +- 100 bash scripts maintained using supervisor I can
get alert for each of them.

For monitoring I am considering using this ->

Any recommendation or advices guys?




Reply via email to