Hi,
Yes, the schema is different, doesn't match with the schema. I think you
need something like this:
Schema (union of records of type "hello1" or "hello2"):
[
{
"type": "record",
"name": "hello1",
"fields": [
{
"name": "foo",
"type": "int",
"default": 1
}
]
},
{
"type": "record",
"name": "hello2",
"fields": [
{
"name": "bar",
"type": "int",
"default": 1
}
]
}
]
If you have a schema like above, you should be able to use hello1 and
hello2 type records:
{"hello1":{"foo": 5}}
{"hello2":{"bar": 10}}
Hope this solves your question.
Regards,
Nandor
On Wed, Jan 10, 2018 at 2:10 PM, kant kodali <[email protected]> wrote:
> Sorry my formatting got messed up. here is the schema I used.
>
>
> {
> "type" : "record",
> "name" : "data",
> "namespace" : "example",
> "fields" : [
> {"type":"record","name":"hello1","fields":[{"name":"foo",
> "type":"int","default":1}]},
> {"type":"record","name":"hello2","fields":[{"name":"bar",
> "type":"int","default":1}]}
> ]
> }
>
>
> On Wed, Jan 10, 2018 at 5:08 AM, kant kodali <[email protected]> wrote:
>
>> Hi Nandor,
>>
>> Thanks a lot for this. What you have said makes logical sense but I am
>> new to Avro so I am just trying to figure out how the schema definition
>> would like for the following messages
>>
>> {"hello1":{"foo": 5}}
>> {"hello2":{"bar": 10}}
>>
>> I have tried the following schema definition to parse the above messages
>> but it didnt quite work so I am wondering how the schema should look like?
>>
>>
>> {
>> "type" : "record",
>> "name" : "data",
>> "namespace" : "example",
>> "fields" : [
>> {"type":"record","name":"hello1","fields":[{"name":"foo",
>> "type":"int","default":1}]},
>> {"type":"record","name":"hello2","fields":[{"name":"bar",
>> "type":"int","default":1}]}
>> ]
>> }
>>
>> On Tue, Jan 9, 2018 at 3:22 AM, Nandor Kollar <[email protected]>
>> wrote:
>>
>>> I think the problem is: you created a union of records, but the Avro
>>> doesn't know if it is a hello1 record instance, or a hello2 record
>>> instance. In this case the you should encode the data like this:
>>> {"hello1":{"foo": 5}}
>>> {"hello2":{"bar": 10}}
>>> Here <https://avro.apache.org/docs/1.8.1/spec.html#json_encoding> is
>>> the relevant part of the specification.
>>>
>>> Nandor
>>>
>>> On Tue, Jan 9, 2018 at 12:06 PM, kant kodali <[email protected]> wrote:
>>>
>>>> Sorry I had a typo I am correcting it here
>>>>
>>>> Hi All,
>>>>
>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>> should be able to parse messages that can either have schema1 or schema2. I
>>>> was thinking to create a union of two records but I am not sure if I am
>>>> doing it right and I am obviously running into various exceptions like
>>>> ArrayOutOfBoundsException and so on.
>>>>
>>>> so I am going to simplify my problem here. Imagine I have the following
>>>> as an Example
>>>>
>>>> *schema1: *
>>>>
>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>> ”:“int”,“default”:1}]}
>>>>
>>>>
>>>> *schema2: *
>>>>
>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>> ”:“int”,“default”:1}]}
>>>>
>>>>
>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I get
>>>> the following
>>>>
>>>> *unionSchema:*
>>>>
>>>> [{“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>> ”:“int”,“default”:1}]},{“type”:“record”,“name”:“hello2”,“
>>>> fields”:[{“name”:“bar”,“type”:“int”,“default”:1}]}]
>>>>
>>>>
>>>>
>>>> Now say my messages inside kafka topic will be something like this
>>>>
>>>> *message1:*
>>>>
>>>> {"foo": 5}
>>>>
>>>> *message2: *
>>>>
>>>> {"bar": 10}
>>>>
>>>>
>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>> why? I can't find any resources on how to do this online. any suggestions
>>>> will be great.
>>>>
>>>> Thanks!
>>>>
>>>> On Tue, Jan 9, 2018 at 3:04 AM, kant kodali <[email protected]> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I have avro messages in a Kafka topic and the requirement is that I
>>>>> should be able to parse messages that can either have schema1 or schema2.
>>>>> I
>>>>> was thinking to create a union of two records but I am not sure if I am
>>>>> doing it right and I am obviously running into various exceptions like
>>>>> ArrayOutOfBoundsException and so on.
>>>>>
>>>>> so I am going to simplify my problem here. Imagine I have the
>>>>> following as an Example
>>>>>
>>>>> *schema1: *
>>>>>
>>>>> {“type”:“record”,“name”:“hello1”,“fields”:[{“name”:“foo”,“type
>>>>> ”:“int”,“default”:1}]}
>>>>>
>>>>>
>>>>> *schema2: *
>>>>>
>>>>> {“type”:“record”,“name”:“hello2”,“fields”:[{“name”:“bar”,“type
>>>>> ”:“int”,“default”:1}]}
>>>>>
>>>>>
>>>>> and if I do Schema.CreateUnion(Arrays.asList(schema1, schema2)) I get
>>>>> the following
>>>>>
>>>>> *unionSchema:*
>>>>>
>>>>> [{“type”:“record”,“name”:“a”,“fields”:[{“name”:“one”,“type”:
>>>>> “int”,“default”:1}]},{“type”:“record”,“name”:“b”,“fields”:[{
>>>>> “name”:“one”,“type”:“int”,“default”:1}]}]
>>>>>
>>>>>
>>>>>
>>>>> Now say my messages inside kafka topic will be something like this
>>>>>
>>>>> *message1:*
>>>>>
>>>>> {"foo": 5}
>>>>>
>>>>> *message2: *
>>>>>
>>>>> {"bar": 10}
>>>>>
>>>>>
>>>>> and if I use unionSchema I am unable to parse it! and I am not sure
>>>>> why? I can't find any resources on how to do this online. any suggestions
>>>>> will be great.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
