Hello,

I'm trying to use a Spark Streaming (1.2.0-cdh5.3.2) consumer
(via spark-streaming-kafka lib of the same version) with Kafka's Confluent
Platform 1.0.

I manage to make a Producer that produce my data and can check it via the
avro-console-consumer :

"./bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181
--from-beginning"

which displays :

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"group":"eeee","domainType":"job","action":"apply_freely","entity":"14564564132","user":{"string":"user"},"session":{"string":"session"},"date":"20150326T154052.000+0100","ip":"192.168.0.1","userAgent":"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/41.0.2272.101 Safari/537.36","referer":"
http://www.kb.com/offre/controleur-de-gestion-h-f-9775029
","os":"Linux","deviceType":"Computer","browser":"Chrome","browserVersion":"41.0.2272.101","browserRenderer":"WEBKIT","physicalServerOrigin":"01.front.local"}
{"group":"eeee","domainType":"job","action":"apply_freely","entity":"14564564132","user":{"string":"user"},"session":{"string":"session"},"date":"20150326T154052.000+0100","ip":"192.168.0.1","userAgent":"Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/41.0.2272.101 Safari/537.36","referer":"
http://www.kb.com/offre/controleur-de-gestion-h-f-9775029
","os":"Linux","deviceType":"Computer","browser":"Chrome","browserVersion":"41.0.2272.101","browserRenderer":"WEBKIT","physicalServerOrigin":"01.front.local"}

So far, so good ! Now here's the weird thing, When I'm using the following
the  spark-streaming-kafka:

val kafkaParams = Map[String, String]("metadata.broker.list" ->
brokers, "group.id" -> consumer, "zookeeper.connect" -> zookeeper,
"auto.offset.reset" -> "smallest", "schema.registry.url" ->
schemaRegistryUrl)
val topicMap = topics.split(",").map((_, 2)).toMap
val messages = KafkaUtils.createStream[Object, Object,
KafkaAvroDecoder, KafkaAvroDecoder](ssc, kafkaParams, topicMap,
StorageLevel.MEMORY_AND_DISK_SER)

val events = messages.map(_._2)
val dStream: DStream[Event] = events.map(
  avroMessage => {
    println("Avro content: " + avroMessage)
  }
)
dStream.print()


Spark Streaming micro batch show me this :


15/04/10 16:11:40 INFO storage.BlockManager: Found block
input-0-1428675099000 locally
Avro content: {"group": "eeee", "domainType": "eeee", "action":
"eeee", "entity": "eeee", "user": "eeee", "session": "eeee", "date":
"eeee", "ip": "eeee", "userAgent": "eeee", "referer": "eeee", "os":
"eeee", "deviceType": "eeee", "browser": "eeee", "browserVersion":
"eeee", "browserRenderer": "eeee", "physicalServerOrigin": "eeee"}
Avro content: {"group": "eeee", "domainType": "eeee", "action":
"eeee", "entity": "eeee", "user": "eeee", "session": "eeee", "date":
"eeee", "ip": "eeee", "userAgent": "eeee", "referer": "eeee", "os":
"eeee", "deviceType": "eeee", "browser": "eeee", "browserVersion":
"eeee", "browserRenderer": "eeee", "physicalServerOrigin": "eeee"}


When I'm changing the snippet code to this (change the StorageLevel to a
non *_SER one) like this :

val messages = KafkaUtils.createStream[Object, Object,
KafkaAvroDecoder, KafkaAvroDecoder](ssc, kafkaParams, topicMap,
*StorageLevel.MEMORY_AND_DISK*)


And it works as expected:


15/04/10 16:29:02 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
15/04/10 16:29:02 INFO storage.BlockManager: Found block
input-0-1428676140400 locally
Avro content: {"group": "eeee", "domainType": "job", "action":
"apply_freely", "entity": "14564564132", "user": "user", "session":
"session", "date": "20150326T154052.000+0100", "ip": "192.168.0.1",
"userAgent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36
(KHTML, like Gecko) Chrome\/41.0.2272.101 Safari\/537.36", "referer":
"http:\/\/www.kb.com\/offre\/controleur-de-gestion-h-f-9775029", "os":
"Linux", "deviceType": "Computer", "browser": "Chrome",
"browserVersion": "41.0.2272.101", "browserRenderer": "WEBKIT",
"physicalServerOrigin": "01.front.local"}
Avro content: {"group": "eeee", "domainType": "job", "action":
"apply_freely", "entity": "14564564132", "user": "user", "session":
"session", "date": "20150326T154052.000+0100", "ip": "192.168.0.1",
"userAgent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36
(KHTML, like Gecko) Chrome\/41.0.2272.101 Safari\/537.36", "referer":
"http:\/\/www.kb.com\/offre\/controleur-de-gestion-h-f-9775029", "os":
"Linux", "deviceType": "Computer", "browser": "Chrome",
"browserVersion": "41.0.2272.101", "browserRenderer": "WEBKIT",
"physicalServerOrigin": "01.front.local"}


If Kryo is disabled, the Java serialization seems to be able to read the
record as expected in both _SER / NON _SER StorageLevel. I've tried several
things with Kryo serialization with no good results (even just activate
Kryo without registering any class manually). I don't know maybe I'm doing
something wrong with the StorageLevel in Spark ?

People in Confluent Platform mailing list
<https://groups.google.com/forum/#!topic/confluent-platform/tNdzyR2Ce00>
suggest me to use chill-avro 0.5.2, so I try the following Kryo
registration with no luck (I've got the same issue described).

kryo.register(classOf[GenericRecord],
AvroSerializer.GenericRecordSerializer[GenericRecord]())
kryo.register(classOf[Event], AvroSerializer.SpecificRecordSerializer[Event])


I've tried with the serializer from ADAMKryoRegistrator.scala
<https://github.com/bigdatagenomics/adam/blob/master/adam-core/src/main/scala/org/bdgenomics/adam/serialization/ADAMKryoRegistrator.scala>
too
but it doesn't work either.

Regards,
Nicolas PHUNG

Reply via email to