Re: How to pass arguments dynamically, that needs to be used in executors

Thanks Todd, that solved my problem. Regards, Gaurav (please excuse spelling mistakes) Sent from phone On Jun 11, 2015 6:42 PM, "Todd Nist" wrote: > Hi Gaurav, > > Seems like you could use a broadcast variable for this if I understand > your use case. Create it in the driver based on the Comman

Re: How to pass arguments dynamically, that needs to be used in executors

Hi Gaurav, Seems like you could use a broadcast variable for this if I understand your use case. Create it in the driver based on the CommandLineArguments and then use it in the workers. https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables So something like: Broadcas

How to pass arguments dynamically, that needs to be used in executors

Hi, I am using Kafka Spark cluster for real time aggregation analytics use case in production. Cluster details 6 nodes, each node running 1 Spark and kafka processes each. Node1 -> 1 Master , 1 Worker, 1 Driver, 1 Kafka process Node 2,3,4,5,6 -> 1 Worker prcocess each