I am not sure if there is a good, clean way to do that - broadcasts variables are not designed to be used out side spark job closures. You could try a bit of a hacky stuff where you write the serialized variable to file in HDFS / NFS / distributed files sytem, and then use a custom decoder class that reads that files and uses it? An even more configurable version would be a custom decoder class that starts a RPC call to the driver running a RPC server to transfer the data.
TD On Fri, Sep 5, 2014 at 10:36 AM, Penny Espinoza < pesp...@societyconsulting.com> wrote: > I need to use a broadcast variable inside the Decoder I use for class > parameter T in org.apache.spark.streaming.kafka.KafkaUtils.createStream. I > am using the override with this signature: > > *createStream > <https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kafka/KafkaUtils.html#createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.util.Map,%20java.util.Map,%20org.apache.spark.storage.StorageLevel)>* > (JavaStreamingContext > <https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/api/java/JavaStreamingContext.html> > jssc, > Class<K> keyTypeClass, Class<V> valueTypeClass, Class<U> keyDecoderClass, > Class<T> valueDecoderClass, java.util.Map<String,String> kafkaParams, > java.util.Map<String,Integer> topics,StorageLevel > <https://spark.apache.org/docs/latest/api/java/org/apache/spark/storage/StorageLevel.html> > storageLevel) > > > Anyone know how I might do that? The actual Decoder instance is > instantiated by Spark, so I don’t know how to access a broadcast variable > inside the fromBytes method. > > thanks > Penny > > > >
