Hi All I have a simple spark application, where I am trying to broadcast a String type variable on YARN Cluster. But every time I am trying to access the broadcast-ed variable value , I am getting null within the Task. It will be really helpful, if you guys can suggest, what I am doing wrong here. My code is like follows:-
public class TestApp implements Serializable{ static Broadcast<String[]> mongoConnectionString; public static void main( String[] args ) { String mongoBaseURL = args[0]; SparkConf sparkConf = new SparkConf().setAppName(Constants.appName); JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf); mongoConnectionString = javaSparkContext.broadcast(args); JavaSQLContext javaSQLContext = new JavaSQLContext(javaSparkContext); JavaSchemaRDD javaSchemaRDD = javaSQLContext.jsonFile(hdfsBaseURL+Constants.hdfsInputDirectoryPath); if(javaSchemaRDD!=null){ javaSchemaRDD.registerTempTable("LogAction"); javaSchemaRDD.cache(); pageSchemaRDD = javaSQLContext.sql(SqlConstants.getLogActionPage); pageSchemaRDD.foreach(new Test()); } } private static class Test implements VoidFunction<Row>{ /** * */ private static final long serialVersionUID = 1L; public void call(Row t) throws Exception { // TODO Auto-generated method stub logger.info("mongoConnectionString "+mongoConnectionString.value()); } } Thanks and Regards Samriddha -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nullpointer-Exception-on-broadcast-variables-YARN-Cluster-mode-tp21929.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org