Re: Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

Mark,

I've attached a simple project to reproduce the bugs, and it fails either
default to 0 or [0]:

Tests in error:
  testIssue1(com.foo.TestIssueDefaultValues): Cannot set the default value
for field [listOfInt] to [0] because that is not a valid value for Data
Type [ARRAY[INT]]
  testIssue2(com.foo.TestIssueDefaultValues): Cannot set the default value
for field [listOfInt] to [[0]] because that is not a valid value for Data
Type [ARRAY[INT]]

So when you said: *Then it works properly, giving us the default value of a
1-element array with the value 0 as the only element*, at least using Nifi
1.5.0 does not work.

Regarding to:
*But what you expect to happenis for this case to be treated the same as if
the schema had said:*

*"type": {"type": "array", "items": "int" }, "default": []*
>>
>>
*So that if the field is not specified, you get an empty array for the
value.*

*Is that accurate? -> **YES*. Basically I guess Nifi should mimic Avro
(when it creates a RecordSchema from Avro schema) in this case in order to
support that scenarios.

Thanks a lot,
Juan

On Tue, 27 Feb 2018 at 11:25 Mark Payne  wrote:

> Juan,
>
> OK, thanks. I was trying to understand what the intent was with that
> schema. So to make sure that everyone
> is on the same page:
>
> If I use the following field in the schema:
>
> "type": {"type": "array", "items": "int" }, "default": [ 0 ]
>>>
>>>
> Then it works properly, giving us the default value of a 1-element array
> with the value 0 as the only element.
>
> But if I use:
>
> "type": {"type": "array", "items": "int" }, "default": 0
>>>
>>>
> Then currently it throws an Exception because the default value is not an
> array. But what you expect to happen
> is for this case to be treated the same as if the schema had said:
>
> "type": {"type": "array", "items": "int" }, "default": []
>>>
>>>
> So that if the field is not specified, you get an empty array for the
> value.
>
> Is that accurate?
>
> Thanks
> -Mark
>
>
> On Feb 27, 2018, at 9:18 AM, Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> Thanks Mike, agree but Avro does not complain and it allows using it. The
> schema is used in production and I cannot change it for now.
>
> On Tue, 27 Feb 2018 at 11:17 Mike Thomsen  wrote:
>
>> That doesn't look like the right way to specify an empty array. This SO
>> example fits about what I'd expect:
>>
>> https://stackoverflow.com/a/42140165/284538
>>
>> So it should be default:[0]
>>
>> On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne  wrote:
>>
>>> Juan,
>>>
>>> So the scenario that you laid out in the NIFI-4893 is not one that I've
>>> personally
>>> encountered. What does it mean exactly to have an Avro schema with an
>>> "array" type
>>> that has a value? In the example that you laid out, the field has:
>>>
>>> "type": {"type": "array", "items": "int" }, "default": 0
>>>
>>>
>>> In this case, what should be the value of this field if it's not
>>> specified? A single-element array with the value of 0?
>>> From looking at the PR that was submitted, it appears to set the
>>> defaultValue to a new (empty) ArrayList.
>>> I would think that maybe it should set defaultValue to
>>>
>>> new Integer[] {0};
>>>
>>> But I am not certain of the semantics here.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella <
>>> gardellajuanpa...@gmail.com> wrote:
>>>
>>> Hello team,
>>>
>>> I could not fix the issue. I did a patch but it's solves the issue, but
>>> I believe is not the correct solution. Anyone that knows Record Framework
>>> can help me on that?
>>>
>>> Thanks in advance,
>>> Juan
>>>
>>> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella <
>>> gardellajuanpa...@gmail.com> wrote:
>>>
 I saw an issue in a test :(. I will continue looking into current
 approach.

 On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
 gardellajuanpa...@gmail.com> wrote:

> Hello team,
>
> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893.
> I discovered using a complex Avro schema. I've isolated the issue and also
> did a patch. At least, this solve the issue but actually I don't know well
> the implications on that solution.
>
> Please let me know what do you think. I have another issue related to
> Avro and Record, I will file the issue tomorrow.
>
> Thanks,
> Juan
>

>>>
>>
>

Re: Cannot convert to Record a valid Avro schema

[Mark Payne]

Juan,

OK, thanks. I was trying to understand what the intent was with that schema. So 
to make sure that everyone
is on the same page:

If I use the following field in the schema:


"type": {"type": "array", "items": "int" }, "default": [ 0 ]

Then it works properly, giving us the default value of a 1-element array with 
the value 0 as the only element.

But if I use:


"type": {"type": "array", "items": "int" }, "default": 0

Then currently it throws an Exception because the default value is not an 
array. But what you expect to happen
is for this case to be treated the same as if the schema had said:


"type": {"type": "array", "items": "int" }, "default": []

So that if the field is not specified, you get an empty array for the value.

Is that accurate?

Thanks
-Mark


On Feb 27, 2018, at 9:18 AM, Juan Pablo Gardella 
> wrote:

Thanks Mike, agree but Avro does not complain and it allows using it. The 
schema is used in production and I cannot change it for now.

On Tue, 27 Feb 2018 at 11:17 Mike Thomsen 
> wrote:
That doesn't look like the right way to specify an empty array. This SO example 
fits about what I'd expect:

https://stackoverflow.com/a/42140165/284538

So it should be default:[0]

On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne 
> wrote:
Juan,

So the scenario that you laid out in the NIFI-4893 is not one that I've 
personally
encountered. What does it mean exactly to have an Avro schema with an "array" 
type
that has a value? In the example that you laid out, the field has:


"type": {"type": "array", "items": "int" }, "default": 0

In this case, what should be the value of this field if it's not specified? A 
single-element array with the value of 0?
>From looking at the PR that was submitted, it appears to set the defaultValue 
>to a new (empty) ArrayList.
I would think that maybe it should set defaultValue to

new Integer[] {0};

But I am not certain of the semantics here.

Thanks
-Mark


On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella 
> wrote:

Hello team,

I could not fix the issue. I did a patch but it's solves the issue, but I 
believe is not the correct solution. Anyone that knows Record Framework can 
help me on that?

Thanks in advance,
Juan

On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella 
> wrote:
I saw an issue in a test :(. I will continue looking into current approach.

On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella 
> wrote:
Hello team,

I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I 
discovered using a complex Avro schema. I've isolated the issue and also did a 
patch. At least, this solve the issue but actually I don't know well the 
implications on that solution.

Please let me know what do you think. I have another issue related to Avro and 
Record, I will file the issue tomorrow.

Thanks,
Juan

Re: Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

Thanks Mike, agree but Avro does not complain and it allows using it. The
schema is used in production and I cannot change it for now.

On Tue, 27 Feb 2018 at 11:17 Mike Thomsen  wrote:

> That doesn't look like the right way to specify an empty array. This SO
> example fits about what I'd expect:
>
> https://stackoverflow.com/a/42140165/284538
>
> So it should be default:[0]
>
> On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne  wrote:
>
>> Juan,
>>
>> So the scenario that you laid out in the NIFI-4893 is not one that I've
>> personally
>> encountered. What does it mean exactly to have an Avro schema with an
>> "array" type
>> that has a value? In the example that you laid out, the field has:
>>
>> "type": {"type": "array", "items": "int" }, "default": 0
>>
>>
>> In this case, what should be the value of this field if it's not
>> specified? A single-element array with the value of 0?
>> From looking at the PR that was submitted, it appears to set the
>> defaultValue to a new (empty) ArrayList.
>> I would think that maybe it should set defaultValue to
>>
>> new Integer[] {0};
>>
>> But I am not certain of the semantics here.
>>
>> Thanks
>> -Mark
>>
>>
>> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella <
>> gardellajuanpa...@gmail.com> wrote:
>>
>> Hello team,
>>
>> I could not fix the issue. I did a patch but it's solves the issue, but I
>> believe is not the correct solution. Anyone that knows Record Framework can
>> help me on that?
>>
>> Thanks in advance,
>> Juan
>>
>> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella <
>> gardellajuanpa...@gmail.com> wrote:
>>
>>> I saw an issue in a test :(. I will continue looking into current
>>> approach.
>>>
>>> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
>>> gardellajuanpa...@gmail.com> wrote:
>>>
 Hello team,

 I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
 discovered using a complex Avro schema. I've isolated the issue and also
 did a patch. At least, this solve the issue but actually I don't know well
 the implications on that solution.

 Please let me know what do you think. I have another issue related to
 Avro and Record, I will file the issue tomorrow.

 Thanks,
 Juan

>>>
>>
>

Re: Cannot convert to Record a valid Avro schema

[Mike Thomsen]

That doesn't look like the right way to specify an empty array. This SO
example fits about what I'd expect:

https://stackoverflow.com/a/42140165/284538

So it should be default:[0]

On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne  wrote:

> Juan,
>
> So the scenario that you laid out in the NIFI-4893 is not one that I've
> personally
> encountered. What does it mean exactly to have an Avro schema with an
> "array" type
> that has a value? In the example that you laid out, the field has:
>
> "type": {"type": "array", "items": "int" }, "default": 0
>
>
> In this case, what should be the value of this field if it's not
> specified? A single-element array with the value of 0?
> From looking at the PR that was submitted, it appears to set the
> defaultValue to a new (empty) ArrayList.
> I would think that maybe it should set defaultValue to
>
> new Integer[] {0};
>
> But I am not certain of the semantics here.
>
> Thanks
> -Mark
>
>
> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> Hello team,
>
> I could not fix the issue. I did a patch but it's solves the issue, but I
> believe is not the correct solution. Anyone that knows Record Framework can
> help me on that?
>
> Thanks in advance,
> Juan
>
> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
>> I saw an issue in a test :(. I will continue looking into current
>> approach.
>>
>> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
>> gardellajuanpa...@gmail.com> wrote:
>>
>>> Hello team,
>>>
>>> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
>>> discovered using a complex Avro schema. I've isolated the issue and also
>>> did a patch. At least, this solve the issue but actually I don't know well
>>> the implications on that solution.
>>>
>>> Please let me know what do you think. I have another issue related to
>>> Avro and Record, I will file the issue tomorrow.
>>>
>>> Thanks,
>>> Juan
>>>
>>
>

Re: Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

Hi Mark,

Thank you for take some time in the issue. Regarding your questions:

* *What does it mean exactly to have an Avro schema with an "array" type
that has a value?* -> A default value I mean.
** What should be the value of this field if it's not specified?* *A
single-element array with the value of 0? * -> Do you mean without default?
It fails with the error: org.apache.avro.AvroRuntimeException: Field
listOfInt type:ARRAY pos:0 not set and has no default value

I agree that the default value is not the best example (I took from a real
scenario), but Avro does not complain and default it to an empty array. I
also added a reproducible scenario with the default value to [0] and also
it fails.

This issue prevent using ConvertRecord because it is not possible to create
a RecordSchema from that Avro schema. I believe, in this case mimic Avro.
If the type is an array and the default value is not an array, default to
an empty array.

Thanks a lot, please let me know if you need something else, I will be glad
to help you. I'm interested in fix of this issue.

Juan

On Tue, 27 Feb 2018 at 10:56 Mark Payne  wrote:

> Juan,
>
> So the scenario that you laid out in the NIFI-4893 is not one that I've
> personally
> encountered. What does it mean exactly to have an Avro schema with an
> "array" type
> that has a value? In the example that you laid out, the field has:
>
> "type": {"type": "array", "items": "int" }, "default": 0
>
>
> In this case, what should be the value of this field if it's not
> specified? A single-element array with the value of 0?
> From looking at the PR that was submitted, it appears to set the
> defaultValue to a new (empty) ArrayList.
> I would think that maybe it should set defaultValue to
>
> new Integer[] {0};
>
> But I am not certain of the semantics here.
>
> Thanks
> -Mark
>
>
> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> Hello team,
>
> I could not fix the issue. I did a patch but it's solves the issue, but I
> believe is not the correct solution. Anyone that knows Record Framework can
> help me on that?
>
> Thanks in advance,
> Juan
>
> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
>> I saw an issue in a test :(. I will continue looking into current
>> approach.
>>
>> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
>> gardellajuanpa...@gmail.com> wrote:
>>
>>> Hello team,
>>>
>>> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
>>> discovered using a complex Avro schema. I've isolated the issue and also
>>> did a patch. At least, this solve the issue but actually I don't know well
>>> the implications on that solution.
>>>
>>> Please let me know what do you think. I have another issue related to
>>> Avro and Record, I will file the issue tomorrow.
>>>
>>> Thanks,
>>> Juan
>>>
>>
>

Re: Cannot convert to Record a valid Avro schema

[Mark Payne]

Juan,

So the scenario that you laid out in the NIFI-4893 is not one that I've 
personally
encountered. What does it mean exactly to have an Avro schema with an "array" 
type
that has a value? In the example that you laid out, the field has:


"type": {"type": "array", "items": "int" }, "default": 0

In this case, what should be the value of this field if it's not specified? A 
single-element array with the value of 0?
>From looking at the PR that was submitted, it appears to set the defaultValue 
>to a new (empty) ArrayList.
I would think that maybe it should set defaultValue to

new Integer[] {0};

But I am not certain of the semantics here.

Thanks
-Mark


On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella 
> wrote:

Hello team,

I could not fix the issue. I did a patch but it's solves the issue, but I 
believe is not the correct solution. Anyone that knows Record Framework can 
help me on that?

Thanks in advance,
Juan

On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella 
> wrote:
I saw an issue in a test :(. I will continue looking into current approach.

On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella 
> wrote:
Hello team,

I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I 
discovered using a complex Avro schema. I've isolated the issue and also did a 
patch. At least, this solve the issue but actually I don't know well the 
implications on that solution.

Please let me know what do you think. I have another issue related to Avro and 
Record, I will file the issue tomorrow.

Thanks,
Juan

Re: Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

Hello team,

I could not fix the issue. I did a patch but it's solves the issue, but I
believe is not the correct solution. Anyone that knows Record Framework can
help me on that?

Thanks in advance,
Juan

On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:

> I saw an issue in a test :(. I will continue looking into current approach.
>
> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
>> Hello team,
>>
>> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
>> discovered using a complex Avro schema. I've isolated the issue and also
>> did a patch. At least, this solve the issue but actually I don't know well
>> the implications on that solution.
>>
>> Please let me know what do you think. I have another issue related to
>> Avro and Record, I will file the issue tomorrow.
>>
>> Thanks,
>> Juan
>>
>

Re: Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

I saw an issue in a test :(. I will continue looking into current approach.

On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:

> Hello team,
>
> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
> discovered using a complex Avro schema. I've isolated the issue and also
> did a patch. At least, this solve the issue but actually I don't know well
> the implications on that solution.
>
> Please let me know what do you think. I have another issue related to Avro
> and Record, I will file the issue tomorrow.
>
> Thanks,
> Juan
>

Cannot convert to Record a valid Avro schema

[Juan Pablo Gardella]

Hello team,

I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I
discovered using a complex Avro schema. I've isolated the issue and also
did a patch. At least, this solve the issue but actually I don't know well
the implications on that solution.

Please let me know what do you think. I have another issue related to Avro
and Record, I will file the issue tomorrow.

Thanks,
Juan