I need to use a broadcast variable inside the Decoder I use for class parameter T in org.apache.spark.streaming.kafka.KafkaUtils.createStream. I am using the override with this signature:
createStream<https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kafka/KafkaUtils.html#createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20java.util.Map,%20java.util.Map,%20org.apache.spark.storage.StorageLevel)>(JavaStreamingContext<https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/api/java/JavaStreamingContext.html> jssc, Class<K> keyTypeClass, Class<V> valueTypeClass, Class<U> keyDecoderClass, Class<T> valueDecoderClass, java.util.Map<String,String> kafkaParams, java.util.Map<String,Integer> topics,StorageLevel<https://spark.apache.org/docs/latest/api/java/org/apache/spark/storage/StorageLevel.html> storageLevel) Anyone know how I might do that? The actual Decoder instance is instantiated by Spark, so I don’t know how to access a broadcast variable inside the fromBytes method. thanks Penny