[
https://issues.apache.org/jira/browse/STORM-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352188#comment-15352188
]
ASF GitHub Bot commented on STORM-1901:
---------------------------------------
Github user vesense commented on a diff in the pull request:
https://github.com/apache/storm/pull/1521#discussion_r68685946
--- Diff:
external/storm-kafka/src/jvm/org/apache/storm/kafka/AvroScheme.java ---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.storm.avro.DefaultDirectAvroSerializer;
+import org.apache.storm.avro.DirectAvroSerializer;
+import org.apache.storm.spout.Scheme;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.Utils;
+
+public class AvroScheme implements Scheme {
+ public static final String AVRO_SCHEME_KEY = "avro";
+ DirectAvroSerializer serializer = new DefaultDirectAvroSerializer();
--- End diff --
>Could this be more flexible, the way AbstractAvroSerializer is above? I
would see creating an AvroSchema and supplying a serializer for it to use based
on how the avro data is being put onto the Kafka topic.
According to my knowledge, `AbstractAvroSerializer` is for the scene
serializing data between workers(due to network communications). We should
register Avro Record to kryo because Kryo dose not support it primitively. This
is why I define a new interface `DirectAvroSerializer` to serialize/deserialize
between avro Record and raw bytes. Also, this is the reason that I separate
Avro from storm-hdfs. Without the avro record registry to kyro, storm-kafka
integration with Avro will get an error like
[STORM-1504](https://issues.apache.org/jira/browse/STORM-1504).
>My original use case was to read avro data that had been serialized onto a
kafka topic with a Confluent serializer. My solution was to create a bolt that
that read the raw data with rawscheme, deserialized the raw bytes into a
GenericRecord, and then emitted the generic record.
I think it is even better we deserialized the raw bytes into GenericRecord
in Spout intead of a bolt.
>Something like this would have been very nice if I could give it a
ConfluentDeserializer via dependency injection or something similar.
Yes, I will update the code to support `ConfluentDeserializer`.
Thanks @dossett for your quick reply. Anything above is wrong please fix
me. Thanks again.
> Avro Integration for Storm-Kafka
> --------------------------------
>
> Key: STORM-1901
> URL: https://issues.apache.org/jira/browse/STORM-1901
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Xin Wang
> Assignee: Xin Wang
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
