what is the best practice to read configure file in spark streaming

[yaoxiaohua]

Hi guys,

                I'm using kafka+spark streaming do log analysis.

                Now my requirement is that the log alarm rules may change
sometimes.

                Rules maybe like this:

                App=Hadoop,keywords=oom|Exception|error,threshold=10

                The threshold or keywords may update sometimes.

                What I do is :

1.       Use a Map[app,logrule] variable to store the log rules. Define it
as a static member.

2.       Use a custom StreamingListener , read the configuration file in
event onBatchStarted

3.       When I use the variable , I found the value is not updated in
windowstream. So now I read the configure file when use

4.       Now I put the log rule in a local path, I should put it in every
worker. 

What 's the best practice to do in this case?

Thanks for your help, I 'm new in spark-streaming , I even not totally
understand the principle.

 

Best Regards,

Evan Yao