Re: Cannot convert to Record a valid Avro schema
Mark, I've attached a simple project to reproduce the bugs, and it fails either default to 0 or [0]: Tests in error: testIssue1(com.foo.TestIssueDefaultValues): Cannot set the default value for field [listOfInt] to [0] because that is not a valid value for Data Type [ARRAY[INT]] testIssue2(com.foo.TestIssueDefaultValues): Cannot set the default value for field [listOfInt] to [[0]] because that is not a valid value for Data Type [ARRAY[INT]] So when you said: *Then it works properly, giving us the default value of a 1-element array with the value 0 as the only element*, at least using Nifi 1.5.0 does not work. Regarding to: *But what you expect to happenis for this case to be treated the same as if the schema had said:* *"type": {"type": "array", "items": "int" }, "default": []* >> >> *So that if the field is not specified, you get an empty array for the value.* *Is that accurate? -> **YES*. Basically I guess Nifi should mimic Avro (when it creates a RecordSchema from Avro schema) in this case in order to support that scenarios. Thanks a lot, Juan On Tue, 27 Feb 2018 at 11:25 Mark Payne wrote: > Juan, > > OK, thanks. I was trying to understand what the intent was with that > schema. So to make sure that everyone > is on the same page: > > If I use the following field in the schema: > > "type": {"type": "array", "items": "int" }, "default": [ 0 ] >>> >>> > Then it works properly, giving us the default value of a 1-element array > with the value 0 as the only element. > > But if I use: > > "type": {"type": "array", "items": "int" }, "default": 0 >>> >>> > Then currently it throws an Exception because the default value is not an > array. But what you expect to happen > is for this case to be treated the same as if the schema had said: > > "type": {"type": "array", "items": "int" }, "default": [] >>> >>> > So that if the field is not specified, you get an empty array for the > value. > > Is that accurate? > > Thanks > -Mark > > > On Feb 27, 2018, at 9:18 AM, Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > > Thanks Mike, agree but Avro does not complain and it allows using it. The > schema is used in production and I cannot change it for now. > > On Tue, 27 Feb 2018 at 11:17 Mike Thomsen wrote: > >> That doesn't look like the right way to specify an empty array. This SO >> example fits about what I'd expect: >> >> https://stackoverflow.com/a/42140165/284538 >> >> So it should be default:[0] >> >> On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne wrote: >> >>> Juan, >>> >>> So the scenario that you laid out in the NIFI-4893 is not one that I've >>> personally >>> encountered. What does it mean exactly to have an Avro schema with an >>> "array" type >>> that has a value? In the example that you laid out, the field has: >>> >>> "type": {"type": "array", "items": "int" }, "default": 0 >>> >>> >>> In this case, what should be the value of this field if it's not >>> specified? A single-element array with the value of 0? >>> From looking at the PR that was submitted, it appears to set the >>> defaultValue to a new (empty) ArrayList. >>> I would think that maybe it should set defaultValue to >>> >>> new Integer[] {0}; >>> >>> But I am not certain of the semantics here. >>> >>> Thanks >>> -Mark >>> >>> >>> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella < >>> gardellajuanpa...@gmail.com> wrote: >>> >>> Hello team, >>> >>> I could not fix the issue. I did a patch but it's solves the issue, but >>> I believe is not the correct solution. Anyone that knows Record Framework >>> can help me on that? >>> >>> Thanks in advance, >>> Juan >>> >>> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella < >>> gardellajuanpa...@gmail.com> wrote: >>> I saw an issue in a test :(. I will continue looking into current approach. On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > Hello team, > > I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. > I discovered using a complex Avro schema. I've isolated the issue and also > did a patch. At least, this solve the issue but actually I don't know well > the implications on that solution. > > Please let me know what do you think. I have another issue related to > Avro and Record, I will file the issue tomorrow. > > Thanks, > Juan > >>> >> >
Re: Cannot convert to Record a valid Avro schema
Juan, OK, thanks. I was trying to understand what the intent was with that schema. So to make sure that everyone is on the same page: If I use the following field in the schema: "type": {"type": "array", "items": "int" }, "default": [ 0 ] Then it works properly, giving us the default value of a 1-element array with the value 0 as the only element. But if I use: "type": {"type": "array", "items": "int" }, "default": 0 Then currently it throws an Exception because the default value is not an array. But what you expect to happen is for this case to be treated the same as if the schema had said: "type": {"type": "array", "items": "int" }, "default": [] So that if the field is not specified, you get an empty array for the value. Is that accurate? Thanks -Mark On Feb 27, 2018, at 9:18 AM, Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: Thanks Mike, agree but Avro does not complain and it allows using it. The schema is used in production and I cannot change it for now. On Tue, 27 Feb 2018 at 11:17 Mike Thomsen mailto:mikerthom...@gmail.com>> wrote: That doesn't look like the right way to specify an empty array. This SO example fits about what I'd expect: https://stackoverflow.com/a/42140165/284538 So it should be default:[0] On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne mailto:marka...@hotmail.com>> wrote: Juan, So the scenario that you laid out in the NIFI-4893 is not one that I've personally encountered. What does it mean exactly to have an Avro schema with an "array" type that has a value? In the example that you laid out, the field has: "type": {"type": "array", "items": "int" }, "default": 0 In this case, what should be the value of this field if it's not specified? A single-element array with the value of 0? >From looking at the PR that was submitted, it appears to set the defaultValue >to a new (empty) ArrayList. I would think that maybe it should set defaultValue to new Integer[] {0}; But I am not certain of the semantics here. Thanks -Mark On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: Hello team, I could not fix the issue. I did a patch but it's solves the issue, but I believe is not the correct solution. Anyone that knows Record Framework can help me on that? Thanks in advance, Juan On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: I saw an issue in a test :(. I will continue looking into current approach. On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: Hello team, I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I discovered using a complex Avro schema. I've isolated the issue and also did a patch. At least, this solve the issue but actually I don't know well the implications on that solution. Please let me know what do you think. I have another issue related to Avro and Record, I will file the issue tomorrow. Thanks, Juan
Re: Cannot convert to Record a valid Avro schema
Thanks Mike, agree but Avro does not complain and it allows using it. The schema is used in production and I cannot change it for now. On Tue, 27 Feb 2018 at 11:17 Mike Thomsen wrote: > That doesn't look like the right way to specify an empty array. This SO > example fits about what I'd expect: > > https://stackoverflow.com/a/42140165/284538 > > So it should be default:[0] > > On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne wrote: > >> Juan, >> >> So the scenario that you laid out in the NIFI-4893 is not one that I've >> personally >> encountered. What does it mean exactly to have an Avro schema with an >> "array" type >> that has a value? In the example that you laid out, the field has: >> >> "type": {"type": "array", "items": "int" }, "default": 0 >> >> >> In this case, what should be the value of this field if it's not >> specified? A single-element array with the value of 0? >> From looking at the PR that was submitted, it appears to set the >> defaultValue to a new (empty) ArrayList. >> I would think that maybe it should set defaultValue to >> >> new Integer[] {0}; >> >> But I am not certain of the semantics here. >> >> Thanks >> -Mark >> >> >> On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella < >> gardellajuanpa...@gmail.com> wrote: >> >> Hello team, >> >> I could not fix the issue. I did a patch but it's solves the issue, but I >> believe is not the correct solution. Anyone that knows Record Framework can >> help me on that? >> >> Thanks in advance, >> Juan >> >> On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella < >> gardellajuanpa...@gmail.com> wrote: >> >>> I saw an issue in a test :(. I will continue looking into current >>> approach. >>> >>> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < >>> gardellajuanpa...@gmail.com> wrote: >>> Hello team, I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I discovered using a complex Avro schema. I've isolated the issue and also did a patch. At least, this solve the issue but actually I don't know well the implications on that solution. Please let me know what do you think. I have another issue related to Avro and Record, I will file the issue tomorrow. Thanks, Juan >>> >> >
Re: Cannot convert to Record a valid Avro schema
That doesn't look like the right way to specify an empty array. This SO example fits about what I'd expect: https://stackoverflow.com/a/42140165/284538 So it should be default:[0] On Tue, Feb 27, 2018 at 8:56 AM, Mark Payne wrote: > Juan, > > So the scenario that you laid out in the NIFI-4893 is not one that I've > personally > encountered. What does it mean exactly to have an Avro schema with an > "array" type > that has a value? In the example that you laid out, the field has: > > "type": {"type": "array", "items": "int" }, "default": 0 > > > In this case, what should be the value of this field if it's not > specified? A single-element array with the value of 0? > From looking at the PR that was submitted, it appears to set the > defaultValue to a new (empty) ArrayList. > I would think that maybe it should set defaultValue to > > new Integer[] {0}; > > But I am not certain of the semantics here. > > Thanks > -Mark > > > On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > > Hello team, > > I could not fix the issue. I did a patch but it's solves the issue, but I > believe is not the correct solution. Anyone that knows Record Framework can > help me on that? > > Thanks in advance, > Juan > > On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > >> I saw an issue in a test :(. I will continue looking into current >> approach. >> >> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < >> gardellajuanpa...@gmail.com> wrote: >> >>> Hello team, >>> >>> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I >>> discovered using a complex Avro schema. I've isolated the issue and also >>> did a patch. At least, this solve the issue but actually I don't know well >>> the implications on that solution. >>> >>> Please let me know what do you think. I have another issue related to >>> Avro and Record, I will file the issue tomorrow. >>> >>> Thanks, >>> Juan >>> >> >
Re: Cannot convert to Record a valid Avro schema
Hi Mark, Thank you for take some time in the issue. Regarding your questions: * *What does it mean exactly to have an Avro schema with an "array" type that has a value?* -> A default value I mean. ** What should be the value of this field if it's not specified?* *A single-element array with the value of 0? * -> Do you mean without default? It fails with the error: org.apache.avro.AvroRuntimeException: Field listOfInt type:ARRAY pos:0 not set and has no default value I agree that the default value is not the best example (I took from a real scenario), but Avro does not complain and default it to an empty array. I also added a reproducible scenario with the default value to [0] and also it fails. This issue prevent using ConvertRecord because it is not possible to create a RecordSchema from that Avro schema. I believe, in this case mimic Avro. If the type is an array and the default value is not an array, default to an empty array. Thanks a lot, please let me know if you need something else, I will be glad to help you. I'm interested in fix of this issue. Juan On Tue, 27 Feb 2018 at 10:56 Mark Payne wrote: > Juan, > > So the scenario that you laid out in the NIFI-4893 is not one that I've > personally > encountered. What does it mean exactly to have an Avro schema with an > "array" type > that has a value? In the example that you laid out, the field has: > > "type": {"type": "array", "items": "int" }, "default": 0 > > > In this case, what should be the value of this field if it's not > specified? A single-element array with the value of 0? > From looking at the PR that was submitted, it appears to set the > defaultValue to a new (empty) ArrayList. > I would think that maybe it should set defaultValue to > > new Integer[] {0}; > > But I am not certain of the semantics here. > > Thanks > -Mark > > > On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > > Hello team, > > I could not fix the issue. I did a patch but it's solves the issue, but I > believe is not the correct solution. Anyone that knows Record Framework can > help me on that? > > Thanks in advance, > Juan > > On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > >> I saw an issue in a test :(. I will continue looking into current >> approach. >> >> On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < >> gardellajuanpa...@gmail.com> wrote: >> >>> Hello team, >>> >>> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I >>> discovered using a complex Avro schema. I've isolated the issue and also >>> did a patch. At least, this solve the issue but actually I don't know well >>> the implications on that solution. >>> >>> Please let me know what do you think. I have another issue related to >>> Avro and Record, I will file the issue tomorrow. >>> >>> Thanks, >>> Juan >>> >> >
Re: Cannot convert to Record a valid Avro schema
Juan, So the scenario that you laid out in the NIFI-4893 is not one that I've personally encountered. What does it mean exactly to have an Avro schema with an "array" type that has a value? In the example that you laid out, the field has: "type": {"type": "array", "items": "int" }, "default": 0 In this case, what should be the value of this field if it's not specified? A single-element array with the value of 0? >From looking at the PR that was submitted, it appears to set the defaultValue >to a new (empty) ArrayList. I would think that maybe it should set defaultValue to new Integer[] {0}; But I am not certain of the semantics here. Thanks -Mark On Feb 27, 2018, at 7:30 AM, Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: Hello team, I could not fix the issue. I did a patch but it's solves the issue, but I believe is not the correct solution. Anyone that knows Record Framework can help me on that? Thanks in advance, Juan On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: I saw an issue in a test :(. I will continue looking into current approach. On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella mailto:gardellajuanpa...@gmail.com>> wrote: Hello team, I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I discovered using a complex Avro schema. I've isolated the issue and also did a patch. At least, this solve the issue but actually I don't know well the implications on that solution. Please let me know what do you think. I have another issue related to Avro and Record, I will file the issue tomorrow. Thanks, Juan
Re: Cannot convert to Record a valid Avro schema
Hello team, I could not fix the issue. I did a patch but it's solves the issue, but I believe is not the correct solution. Anyone that knows Record Framework can help me on that? Thanks in advance, Juan On Mon, 19 Feb 2018 at 22:33 Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > I saw an issue in a test :(. I will continue looking into current approach. > > On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > >> Hello team, >> >> I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I >> discovered using a complex Avro schema. I've isolated the issue and also >> did a patch. At least, this solve the issue but actually I don't know well >> the implications on that solution. >> >> Please let me know what do you think. I have another issue related to >> Avro and Record, I will file the issue tomorrow. >> >> Thanks, >> Juan >> >
Re: Cannot convert to Record a valid Avro schema
I saw an issue in a test :(. I will continue looking into current approach. On Mon, 19 Feb 2018 at 22:23 Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > Hello team, > > I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I > discovered using a complex Avro schema. I've isolated the issue and also > did a patch. At least, this solve the issue but actually I don't know well > the implications on that solution. > > Please let me know what do you think. I have another issue related to Avro > and Record, I will file the issue tomorrow. > > Thanks, > Juan >
Cannot convert to Record a valid Avro schema
Hello team, I filed an issue at https://issues.apache.org/jira/browse/NIFI-4893. I discovered using a complex Avro schema. I've isolated the issue and also did a patch. At least, this solve the issue but actually I don't know well the implications on that solution. Please let me know what do you think. I have another issue related to Avro and Record, I will file the issue tomorrow. Thanks, Juan