Broadcast variables: when should I use them?

2015-01-26 Thread frodo777
Hello. I have a number of static Arrays and Maps in my Spark Streaming driver program. They are simple collections, initialized with integer values and strings directly in the code. There is no RDD/DStream involvement here. I do not expect them to contain more than 100 entries, each. They are

Re: Spark Standalone Cluster not correctly configured

2015-01-08 Thread frodo777
Hello everyone. With respect to the configuration problem that I explained before Do you have any idea what is wrong there? The problem in a nutshell: - When more than one master is started in the cluster, all of them are scheduling independently, thinking they are all leaders. - zookeeper

Spark Standalone Cluster not correctly configured

2014-12-30 Thread frodo777
Hi. I'm trying to configure a spark standalone cluster, with three master nodes (bigdata1, bigdata2 and bigdata3) managed by Zookeeper. It seems there's a configuration problem, since everyone is saying it is the cluster leader: . 14/12/30 13:54:59 INFO Master: I have been