Thanks Nandor! Which schema should I use for serialization now? say I want
to serialize the following
{"hello1":{"foo": 5}}
On Wed, Jan 10, 2018 at 6:05 AM, Nandor Kollar <[email protected]> wrote:
> Hi,
>
> Yes, the schema is different, doesn't match with the schema. I think you
> need something like this:
>
> Schema (union of records of type "hello1" or "hello2"):
>
> [
> {
> "type": "record",
> "name": "hello1",
> "fields": [
> {
> "name": "foo",
> "type": "int",
> "default": 1
> }
> ]
> },
> {
> "type": "record",
> "name": "hello2",
> "fields": [
> {
> "name": "bar",
> "type": "int",
> "default": 1
> }
> ]
> }
> ]
>
> If you have a schema like above, you should be able to use hello1 and
> hello2 type records:
> {"hello1":{"foo": 5}}
> {"hello2":{"bar": 10}}
>
> Hope this solves your question.
>
> Regards,
> Nandor
>
> On Wed, Jan 10, 2018 at 2:10 PM, kant kodali <[email protected]> wrote:
>
>> Sorry my formatting got messed up. here is the schema I used.
>>
>>
>> {
>> "type" : "record",
>> "name" : "data",
>> "namespace" : "example",
>> "fields" : [
>> {"type":"record","name":"hello1","fields":[{"name":"foo"
>> ,"type":"int","default":1}]},
>> {"type":"record","name":"hello2","fields":[{"name":"bar"
>> ,"type":"int","default":1}]}
>> ]
>> }
>>
>>
>> On Wed, Jan 10, 2018 at 5:08 AM, kant kodali <[email protected]> wrote:
>>
>>> Hi Nandor,
>>>
>>> Thanks a lot for this. What you have said makes logical sense but I am
>>> new to Avro so I am just trying to figure out how the schema definition
>>> would like for the following messages
>>>
>>> {"hello1":{"foo": 5}}
>>> {"hello2":{"bar": 10}}
>>>
>>> I have tried the following schema definition to parse the above messages
>>> but it didnt quite work so I am wondering how the schema should look like?
>>>
>>>
>>> {
>>> "type" : "record",
>>> "name" : "data",
>>> "namespace" : "example",
>>> "fields" : [
>>> {"type":"record","name":"hello1","fields":[{"name":"foo"
>>> ,"type":"int","default":1}]},
>>> {"type":"record","name":"hello2","fields":[{"name":"bar"
>>> ,"type":"int","default":1}]}
>>> ]
>>> }
>>>
>>> On Tue, Jan 9, 2018 at 3:22 AM, Nandor Kollar <[email protected]>
>>> wrote:
>>>
>>>> I think the problem is: you created a union of records, but the Avro
>>>> doesn't know if it is a hello1 record instance, or a hello2 record
>>>> instance. In this case the you should encode the data like this:
>>>> {"hello1":{"foo": 5}}
>>>> {"hello2":{"bar": 10}}
>>>> Here <https://avro.apache.org/docs/1.8.1/spec.html#json_encoding> is
>>>> the relevant part of the specification.
>>>>
>>>> Nandor
>>>>
>>>> On Tue, Jan 9, 2018 at 12:06 PM, kant kodali <[email protected]>
>>>> wrote:
>>>>
>>>>> Sorry I had a typo I am correcting it here
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>>> should be able to parse messages that can either have schema1 or schema2.
>>>>> I
>>>>> was thinking to create a union of two records but I am not sure if I am
>>>>> doing it right and I am obviously running into various exceptions like
>>>>> ArrayOutOfBoundsException and so on.
>>>>>
>>>>> so I am going to simplify my problem here. Imagine I have the
>>>>> following as an Example
>>>>>
>>>>> *schema1: *
>>>>>
>>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>> ”:“int”,“default”:1}]}
>>>>>
>>>>>
>>>>> *schema2: *
>>>>>
>>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>>> ”:“int”,“default”:1}]}
>>>>>
>>>>>
>>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I get
>>>>> the following
>>>>>
>>>>> *unionSchema:*
>>>>>
>>>>> [{“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>> ”:“int”,“default”:1}]},{“type”:“record”,“name”:“hello2”,“
>>>>> fields”:[{“name”:“bar”,“type”:“int”,“default”:1}]}]
>>>>>
>>>>>
>>>>>
>>>>> Now say my messages inside kafka topic will be something like this
>>>>>
>>>>> *message1:*
>>>>>
>>>>> {"foo": 5}
>>>>>
>>>>> *message2: *
>>>>>
>>>>> {"bar": 10}
>>>>>
>>>>>
>>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>>> why? I can't find any resources on how to do this online. any suggestions
>>>>> will be great.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Tue, Jan 9, 2018 at 3:04 AM, kant kodali <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>>>> should be able to parse messages that can either have schema1 or
>>>>>> schema2. I
>>>>>> was thinking to create a union of two records but I am not sure if I am
>>>>>> doing it right and I am obviously running into various exceptions like
>>>>>> ArrayOutOfBoundsException and so on.
>>>>>>
>>>>>> so I am going to simplify my problem here. Imagine I have the
>>>>>> following as an Example
>>>>>>
>>>>>> *schema1: *
>>>>>>
>>>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>>> ”:“int”,“default”:1}]}
>>>>>>
>>>>>>
>>>>>> *schema2: *
>>>>>>
>>>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>>>> ”:“int”,“default”:1}]}
>>>>>>
>>>>>>
>>>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I
>>>>>> get the following
>>>>>>
>>>>>> *unionSchema:*
>>>>>>
>>>>>> [{“type”:“record”,“name”:“a”,“fields”:[{“name”:“one”,“type”:
>>>>>> “int”,“default”:1}]},{“type”:“record”,“name”:“b”,“fields”:[{
>>>>>> “name”:“one”,“type”:“int”,“default”:1}]}]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now say my messages inside kafka topic will be something like this
>>>>>>
>>>>>> *message1:*
>>>>>>
>>>>>> {"foo": 5}
>>>>>>
>>>>>> *message2: *
>>>>>>
>>>>>> {"bar": 10}
>>>>>>
>>>>>>
>>>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>>>> why? I can't find any resources on how to do this online. any suggestions
>>>>>> will be great.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
