Hello, I'm trying to use a Spark Streaming (1.2.0-cdh5.3.2) consumer (via spark-streaming-kafka lib of the same version) with Kafka's Confluent Platform 1.0.
I manage to make a Producer that produce my data and can check it via the avro-console-consumer : "./bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181 --from-beginning" which displays : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] {"group":"eeee","domainType":"job","action":"apply_freely","entity":"14564564132","user":{"string":"user"},"session":{"string":"session"},"date":"20150326T154052.000+0100","ip":"192.168.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36","referer":" http://www.kb.com/offre/controleur-de-gestion-h-f-9775029 ","os":"Linux","deviceType":"Computer","browser":"Chrome","browserVersion":"41.0.2272.101","browserRenderer":"WEBKIT","physicalServerOrigin":"01.front.local"} {"group":"eeee","domainType":"job","action":"apply_freely","entity":"14564564132","user":{"string":"user"},"session":{"string":"session"},"date":"20150326T154052.000+0100","ip":"192.168.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36","referer":" http://www.kb.com/offre/controleur-de-gestion-h-f-9775029 ","os":"Linux","deviceType":"Computer","browser":"Chrome","browserVersion":"41.0.2272.101","browserRenderer":"WEBKIT","physicalServerOrigin":"01.front.local"} So far, so good ! Now here's the weird thing, When I'm using the following the spark-streaming-kafka: val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers, "group.id" -> consumer, "zookeeper.connect" -> zookeeper, "auto.offset.reset" -> "smallest", "schema.registry.url" -> schemaRegistryUrl) val topicMap = topics.split(",").map((_, 2)).toMap val messages = KafkaUtils.createStream[Object, Object, KafkaAvroDecoder, KafkaAvroDecoder](ssc, kafkaParams, topicMap, StorageLevel.MEMORY_AND_DISK_SER) val events = messages.map(_._2) val dStream: DStream[Event] = events.map( avroMessage => { println("Avro content: " + avroMessage) } ) dStream.print() Spark Streaming micro batch show me this : 15/04/10 16:11:40 INFO storage.BlockManager: Found block input-0-1428675099000 locally Avro content: {"group": "eeee", "domainType": "eeee", "action": "eeee", "entity": "eeee", "user": "eeee", "session": "eeee", "date": "eeee", "ip": "eeee", "userAgent": "eeee", "referer": "eeee", "os": "eeee", "deviceType": "eeee", "browser": "eeee", "browserVersion": "eeee", "browserRenderer": "eeee", "physicalServerOrigin": "eeee"} Avro content: {"group": "eeee", "domainType": "eeee", "action": "eeee", "entity": "eeee", "user": "eeee", "session": "eeee", "date": "eeee", "ip": "eeee", "userAgent": "eeee", "referer": "eeee", "os": "eeee", "deviceType": "eeee", "browser": "eeee", "browserVersion": "eeee", "browserRenderer": "eeee", "physicalServerOrigin": "eeee"} When I'm changing the snippet code to this (change the StorageLevel to a non *_SER one) like this : val messages = KafkaUtils.createStream[Object, Object, KafkaAvroDecoder, KafkaAvroDecoder](ssc, kafkaParams, topicMap, *StorageLevel.MEMORY_AND_DISK*) And it works as expected: 15/04/10 16:29:02 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2) 15/04/10 16:29:02 INFO storage.BlockManager: Found block input-0-1428676140400 locally Avro content: {"group": "eeee", "domainType": "job", "action": "apply_freely", "entity": "14564564132", "user": "user", "session": "session", "date": "20150326T154052.000+0100", "ip": "192.168.0.1", "userAgent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.101 Safari\/537.36", "referer": "http:\/\/www.kb.com\/offre\/controleur-de-gestion-h-f-9775029", "os": "Linux", "deviceType": "Computer", "browser": "Chrome", "browserVersion": "41.0.2272.101", "browserRenderer": "WEBKIT", "physicalServerOrigin": "01.front.local"} Avro content: {"group": "eeee", "domainType": "job", "action": "apply_freely", "entity": "14564564132", "user": "user", "session": "session", "date": "20150326T154052.000+0100", "ip": "192.168.0.1", "userAgent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.101 Safari\/537.36", "referer": "http:\/\/www.kb.com\/offre\/controleur-de-gestion-h-f-9775029", "os": "Linux", "deviceType": "Computer", "browser": "Chrome", "browserVersion": "41.0.2272.101", "browserRenderer": "WEBKIT", "physicalServerOrigin": "01.front.local"} If Kryo is disabled, the Java serialization seems to be able to read the record as expected in both _SER / NON _SER StorageLevel. I've tried several things with Kryo serialization with no good results (even just activate Kryo without registering any class manually). I don't know maybe I'm doing something wrong with the StorageLevel in Spark ? People in Confluent Platform mailing list <https://groups.google.com/forum/#!topic/confluent-platform/tNdzyR2Ce00> suggest me to use chill-avro 0.5.2, so I try the following Kryo registration with no luck (I've got the same issue described). kryo.register(classOf[GenericRecord], AvroSerializer.GenericRecordSerializer[GenericRecord]()) kryo.register(classOf[Event], AvroSerializer.SpecificRecordSerializer[Event]) I've tried with the serializer from ADAMKryoRegistrator.scala <https://github.com/bigdatagenomics/adam/blob/master/adam-core/src/main/scala/org/bdgenomics/adam/serialization/ADAMKryoRegistrator.scala> too but it doesn't work either. Regards, Nicolas PHUNG
