Hi Nandor,
On an another note Is there a way to come up with one schema that can work
for both of these messages?
{"hello1":{"foo": 5}}
{"foo": 5}
Thanks a lot!
On Wed, Jan 10, 2018 at 8:29 AM, kant kodali <[email protected]> wrote:
> Thanks Nandor! Which schema should I use for serialization now? say I want
> to serialize the following
>
> {"hello1":{"foo": 5}}
>
> On Wed, Jan 10, 2018 at 6:05 AM, Nandor Kollar <[email protected]>
> wrote:
>
>> Hi,
>>
>> Yes, the schema is different, doesn't match with the schema. I think you
>> need something like this:
>>
>> Schema (union of records of type "hello1" or "hello2"):
>>
>> [
>> {
>> "type": "record",
>> "name": "hello1",
>> "fields": [
>> {
>> "name": "foo",
>> "type": "int",
>> "default": 1
>> }
>> ]
>> },
>> {
>> "type": "record",
>> "name": "hello2",
>> "fields": [
>> {
>> "name": "bar",
>> "type": "int",
>> "default": 1
>> }
>> ]
>> }
>> ]
>>
>> If you have a schema like above, you should be able to use hello1 and
>> hello2 type records:
>> {"hello1":{"foo": 5}}
>> {"hello2":{"bar": 10}}
>>
>> Hope this solves your question.
>>
>> Regards,
>> Nandor
>>
>> On Wed, Jan 10, 2018 at 2:10 PM, kant kodali <[email protected]> wrote:
>>
>>> Sorry my formatting got messed up. here is the schema I used.
>>>
>>>
>>> {
>>> "type" : "record",
>>> "name" : "data",
>>> "namespace" : "example",
>>> "fields" : [
>>> {"type":"record","name":"hello1","fields":[{"name":
>>> "foo","type":"int","default":1}]},
>>> {"type":"record","name":"hello2","fields":[{"name":
>>> "bar","type":"int","default":1}]}
>>> ]
>>> }
>>>
>>>
>>> On Wed, Jan 10, 2018 at 5:08 AM, kant kodali <[email protected]> wrote:
>>>
>>>> Hi Nandor,
>>>>
>>>> Thanks a lot for this. What you have said makes logical sense but I am
>>>> new to Avro so I am just trying to figure out how the schema definition
>>>> would like for the following messages
>>>>
>>>> {"hello1":{"foo": 5}}
>>>> {"hello2":{"bar": 10}}
>>>>
>>>> I have tried the following schema definition to parse the above
>>>> messages but it didnt quite work so I am wondering how the schema should
>>>> look like?
>>>>
>>>>
>>>> {
>>>> "type" : "record",
>>>> "name" : "data",
>>>> "namespace" : "example",
>>>> "fields" : [
>>>> {"type":"record","name":"hello1","fields":[{"name":
>>>> "foo","type":"int","default":1}]},
>>>> {"type":"record","name":"hello2","fields":[{"name":
>>>> "bar","type":"int","default":1}]}
>>>> ]
>>>> }
>>>>
>>>> On Tue, Jan 9, 2018 at 3:22 AM, Nandor Kollar <[email protected]>
>>>> wrote:
>>>>
>>>>> I think the problem is: you created a union of records, but the Avro
>>>>> doesn't know if it is a hello1 record instance, or a hello2 record
>>>>> instance. In this case the you should encode the data like this:
>>>>> {"hello1":{"foo": 5}}
>>>>> {"hello2":{"bar": 10}}
>>>>> Here <https://avro.apache.org/docs/1.8.1/spec.html#json_encoding> is
>>>>> the relevant part of the specification.
>>>>>
>>>>> Nandor
>>>>>
>>>>> On Tue, Jan 9, 2018 at 12:06 PM, kant kodali <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Sorry I had a typo I am correcting it here
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>>>> should be able to parse messages that can either have schema1 or
>>>>>> schema2. I
>>>>>> was thinking to create a union of two records but I am not sure if I am
>>>>>> doing it right and I am obviously running into various exceptions like
>>>>>> ArrayOutOfBoundsException and so on.
>>>>>>
>>>>>> so I am going to simplify my problem here. Imagine I have the
>>>>>> following as an Example
>>>>>>
>>>>>> *schema1: *
>>>>>>
>>>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>>> ”:“int”,“default”:1}]}
>>>>>>
>>>>>>
>>>>>> *schema2: *
>>>>>>
>>>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>>>> ”:“int”,“default”:1}]}
>>>>>>
>>>>>>
>>>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I
>>>>>> get the following
>>>>>>
>>>>>> *unionSchema:*
>>>>>>
>>>>>> [{“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>>> ”:“int”,“default”:1}]},{“type”:“record”,“name”:“hello2”,“
>>>>>> fields”:[{“name”:“bar”,“type”:“int”,“default”:1}]}]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now say my messages inside kafka topic will be something like this
>>>>>>
>>>>>> *message1:*
>>>>>>
>>>>>> {"foo": 5}
>>>>>>
>>>>>> *message2: *
>>>>>>
>>>>>> {"bar": 10}
>>>>>>
>>>>>>
>>>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>>>> why? I can't find any resources on how to do this online. any suggestions
>>>>>> will be great.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Tue, Jan 9, 2018 at 3:04 AM, kant kodali <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>>>>> should be able to parse messages that can either have schema1 or
>>>>>>> schema2. I
>>>>>>> was thinking to create a union of two records but I am not sure if I am
>>>>>>> doing it right and I am obviously running into various exceptions like
>>>>>>> ArrayOutOfBoundsException and so on.
>>>>>>>
>>>>>>> so I am going to simplify my problem here. Imagine I have the
>>>>>>> following as an Example
>>>>>>>
>>>>>>> *schema1: *
>>>>>>>
>>>>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>>>> ”:“int”,“default”:1}]}
>>>>>>>
>>>>>>>
>>>>>>> *schema2: *
>>>>>>>
>>>>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>>>>> ”:“int”,“default”:1}]}
>>>>>>>
>>>>>>>
>>>>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I
>>>>>>> get the following
>>>>>>>
>>>>>>> *unionSchema:*
>>>>>>>
>>>>>>> [{“type”:“record”,“name”:“a”,“fields”:[{“name”:“one”,“type”:
>>>>>>> “int”,“default”:1}]},{“type”:“record”,“name”:“b”,“fields”:[{
>>>>>>> “name”:“one”,“type”:“int”,“default”:1}]}]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Now say my messages inside kafka topic will be something like this
>>>>>>>
>>>>>>> *message1:*
>>>>>>>
>>>>>>> {"foo": 5}
>>>>>>>
>>>>>>> *message2: *
>>>>>>>
>>>>>>> {"bar": 10}
>>>>>>>
>>>>>>>
>>>>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>>>>> why? I can't find any resources on how to do this online. any
>>>>>>> suggestions
>>>>>>> will be great.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
