Re: Issue with ValidateRecord processor and Avro map data type
I believe it's a bug too. Thank you Matt for opening the issue !
On Wed, Oct 10, 2018 at 5:15 PM Matt Burgess wrote:
> Maxime,
>
> I believe this is a bug, specifically at [1]. It should support
> MapRecord entries as well. I have written NIFI-5678 [2] to address
> this. The workaround is the one you and Bryan mentioned, using a
> record instead of a map.
>
> Regards,
> Matt
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java#L199
> [2] https://issues.apache.org/jira/browse/NIFI-5678
>
> On Wed, Oct 10, 2018 at 10:19 AM Maxime Jattiot
> wrote:
> >
> > Yes I made it worked with record type as well but I can't use a record
> > because my field names are dynamic.
> >
> > In many JSON we use collections of keys that will be named at runtime. I
> > thought the map data type of Avro was specially designed for it
> >
> > Any other ideas ?
> >
> >
> > On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende wrote:
> >
> > > I was able to make it work using a nested record instead of a map
> > >
> > > {
> > > "type": "record",
> > > "name": "example",
> > > "fields" : [
> > > {
> > > "name": "field1",
> > > "type": {
> > > "name" : "field1",
> > > "type" : "record",
> > > "fields" : [
> > > { "name": "toto", "type": "string" },
> > > { "name": "titi", "type": "string" }
> > > ]
> > > }
> > > }
> > > ]
> > > }
> > > On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot >
> > > wrote:
> > > >
> > > > Sure, here is the schema :
> > > > {
> > > > "name" : "test",
> > > > "type" : "record",
> > > > "fields" : [ {
> > > > "name" : "field1",
> > > > "type" : {
> > > > "type" : "map",
> > > > "values" : "string"
> > > > }
> > > > } ]
> > > > }
> > > >
> > > > And here is the JSON message :
> > > > [{
> > > > "field1": {
> > > > "toto" : "v1",
> > > > "titi" : "v2"
> > > > }
> > > > }]
> > > >
> > > > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > The screenshots did not come through, maybe just paste the schema
> and
> > > JSON
> > > > > as text in the email.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <
> mjatt...@opensense.fr>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I am having hard time to validate a simple JSON message with an
> Avro
> > > > > > schema. My schema works with avro-tools but doesn't with
> > > ValidateRecord
> > > > > > processor. I wish you could help me.
> > > > > >
> > > > > >- I am having the following debug log within Nifi :
> > > > > >
> > > > > > Record #1 is invalid due to:
> > > > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> > > Value is
> > > > > > of type org.apache.nifi.serialization.record.MapRecord but was
> > > expected
> > > > > to
> > > > > > be of type MAP[STRING]
> > > > > >
> > > > > >
> > > > > >- Here is my schema :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > > > >
> > > > > >- Here is the JSON content sent :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > > > >
> > > > > > Does someone has any clue ? I have been struggling 2 days with
> this
> > > map
> > > > > > data type ... :(
> > > > > >
> > > > > > Thank you !
> > > > > >
> > > > > > Maxime
> > > > > >
> > > > >
> > >
>
Re: Issue with ValidateRecord processor and Avro map data type
Maxime,
I believe this is a bug, specifically at [1]. It should support
MapRecord entries as well. I have written NIFI-5678 [2] to address
this. The workaround is the one you and Bryan mentioned, using a
record instead of a map.
Regards,
Matt
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java#L199
[2] https://issues.apache.org/jira/browse/NIFI-5678
On Wed, Oct 10, 2018 at 10:19 AM Maxime Jattiot wrote:
>
> Yes I made it worked with record type as well but I can't use a record
> because my field names are dynamic.
>
> In many JSON we use collections of keys that will be named at runtime. I
> thought the map data type of Avro was specially designed for it
>
> Any other ideas ?
>
>
> On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende wrote:
>
> > I was able to make it work using a nested record instead of a map
> >
> > {
> > "type": "record",
> > "name": "example",
> > "fields" : [
> > {
> > "name": "field1",
> > "type": {
> > "name" : "field1",
> > "type" : "record",
> > "fields" : [
> > { "name": "toto", "type": "string" },
> > { "name": "titi", "type": "string" }
> > ]
> > }
> > }
> > ]
> > }
> > On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot
> > wrote:
> > >
> > > Sure, here is the schema :
> > > {
> > > "name" : "test",
> > > "type" : "record",
> > > "fields" : [ {
> > > "name" : "field1",
> > > "type" : {
> > > "type" : "map",
> > > "values" : "string"
> > > }
> > > } ]
> > > }
> > >
> > > And here is the JSON message :
> > > [{
> > > "field1": {
> > > "toto" : "v1",
> > > "titi" : "v2"
> > > }
> > > }]
> > >
> > > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende wrote:
> > >
> > > > Hello,
> > > >
> > > > The screenshots did not come through, maybe just paste the schema and
> > JSON
> > > > as text in the email.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I am having hard time to validate a simple JSON message with an Avro
> > > > > schema. My schema works with avro-tools but doesn't with
> > ValidateRecord
> > > > > processor. I wish you could help me.
> > > > >
> > > > >- I am having the following debug log within Nifi :
> > > > >
> > > > > Record #1 is invalid due to:
> > > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> > Value is
> > > > > of type org.apache.nifi.serialization.record.MapRecord but was
> > expected
> > > > to
> > > > > be of type MAP[STRING]
> > > > >
> > > > >
> > > > >- Here is my schema :
> > > > >
> > > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > > >
> > > > >- Here is the JSON content sent :
> > > > >
> > > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > > >
> > > > > Does someone has any clue ? I have been struggling 2 days with this
> > map
> > > > > data type ... :(
> > > > >
> > > > > Thank you !
> > > > >
> > > > > Maxime
> > > > >
> > > >
> >
Re: Issue with ValidateRecord processor and Avro map data type
Yes I made it worked with record type as well but I can't use a record
because my field names are dynamic.
In many JSON we use collections of keys that will be named at runtime. I
thought the map data type of Avro was specially designed for it
Any other ideas ?
On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende wrote:
> I was able to make it work using a nested record instead of a map
>
> {
> "type": "record",
> "name": "example",
> "fields" : [
> {
> "name": "field1",
> "type": {
> "name" : "field1",
> "type" : "record",
> "fields" : [
> { "name": "toto", "type": "string" },
> { "name": "titi", "type": "string" }
> ]
> }
> }
> ]
> }
> On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot
> wrote:
> >
> > Sure, here is the schema :
> > {
> > "name" : "test",
> > "type" : "record",
> > "fields" : [ {
> > "name" : "field1",
> > "type" : {
> > "type" : "map",
> > "values" : "string"
> > }
> > } ]
> > }
> >
> > And here is the JSON message :
> > [{
> > "field1": {
> > "toto" : "v1",
> > "titi" : "v2"
> > }
> > }]
> >
> > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende wrote:
> >
> > > Hello,
> > >
> > > The screenshots did not come through, maybe just paste the schema and
> JSON
> > > as text in the email.
> > >
> > > Thanks.
> > >
> > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I am having hard time to validate a simple JSON message with an Avro
> > > > schema. My schema works with avro-tools but doesn't with
> ValidateRecord
> > > > processor. I wish you could help me.
> > > >
> > > >- I am having the following debug log within Nifi :
> > > >
> > > > Record #1 is invalid due to:
> > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> Value is
> > > > of type org.apache.nifi.serialization.record.MapRecord but was
> expected
> > > to
> > > > be of type MAP[STRING]
> > > >
> > > >
> > > >- Here is my schema :
> > > >
> > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > >
> > > >- Here is the JSON content sent :
> > > >
> > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > >
> > > > Does someone has any clue ? I have been struggling 2 days with this
> map
> > > > data type ... :(
> > > >
> > > > Thank you !
> > > >
> > > > Maxime
> > > >
> > >
>
Re: Issue with ValidateRecord processor and Avro map data type
I was able to make it work using a nested record instead of a map
{
"type": "record",
"name": "example",
"fields" : [
{
"name": "field1",
"type": {
"name" : "field1",
"type" : "record",
"fields" : [
{ "name": "toto", "type": "string" },
{ "name": "titi", "type": "string" }
]
}
}
]
}
On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot wrote:
>
> Sure, here is the schema :
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
>
> And here is the JSON message :
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
>
> On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende wrote:
>
> > Hello,
> >
> > The screenshots did not come through, maybe just paste the schema and JSON
> > as text in the email.
> >
> > Thanks.
> >
> > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I am having hard time to validate a simple JSON message with an Avro
> > > schema. My schema works with avro-tools but doesn't with ValidateRecord
> > > processor. I wish you could help me.
> > >
> > >- I am having the following debug log within Nifi :
> > >
> > > Record #1 is invalid due to:
> > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> > > of type org.apache.nifi.serialization.record.MapRecord but was expected
> > to
> > > be of type MAP[STRING]
> > >
> > >
> > >- Here is my schema :
> > >
> > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > >
> > >- Here is the JSON content sent :
> > >
> > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > >
> > > Does someone has any clue ? I have been struggling 2 days with this map
> > > data type ... :(
> > >
> > > Thank you !
> > >
> > > Maxime
> > >
> >
Re: Issue with ValidateRecord processor and Avro map data type
Sure, here is the schema :
{
"name" : "test",
"type" : "record",
"fields" : [ {
"name" : "field1",
"type" : {
"type" : "map",
"values" : "string"
}
} ]
}
And here is the JSON message :
[{
"field1": {
"toto" : "v1",
"titi" : "v2"
}
}]
On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende wrote:
> Hello,
>
> The screenshots did not come through, maybe just paste the schema and JSON
> as text in the email.
>
> Thanks.
>
> On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot
> wrote:
>
> > Hello everyone,
> >
> > I am having hard time to validate a simple JSON message with an Avro
> > schema. My schema works with avro-tools but doesn't with ValidateRecord
> > processor. I wish you could help me.
> >
> >- I am having the following debug log within Nifi :
> >
> > Record #1 is invalid due to:
> > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> > of type org.apache.nifi.serialization.record.MapRecord but was expected
> to
> > be of type MAP[STRING]
> >
> >
> >- Here is my schema :
> >
> > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> >
> >- Here is the JSON content sent :
> >
> > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> >
> > Does someone has any clue ? I have been struggling 2 days with this map
> > data type ... :(
> >
> > Thank you !
> >
> > Maxime
> >
>
Re: Issue with ValidateRecord processor and Avro map data type
Hello,
The screenshots did not come through, maybe just paste the schema and JSON
as text in the email.
Thanks.
On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot
wrote:
> Hello everyone,
>
> I am having hard time to validate a simple JSON message with an Avro
> schema. My schema works with avro-tools but doesn't with ValidateRecord
> processor. I wish you could help me.
>
>- I am having the following debug log within Nifi :
>
> Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> of type org.apache.nifi.serialization.record.MapRecord but was expected to
> be of type MAP[STRING]
>
>
>- Here is my schema :
>
> [image: Screen Shot 2018-10-10 at 11.21.41.png]
>
>- Here is the JSON content sent :
>
> [image: Screen Shot 2018-10-10 at 11.22.10.png]
>
> Does someone has any clue ? I have been struggling 2 days with this map
> data type ... :(
>
> Thank you !
>
> Maxime
>
Issue with ValidateRecord processor and Avro map data type
Hello everyone,
I am having hard time to validate a simple JSON message with an Avro
schema. My schema works with avro-tools but doesn't with ValidateRecord
processor. I wish you could help me.
- I am having the following debug log within Nifi :
Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
type org.apache.nifi.serialization.record.MapRecord but was expected to be
of type MAP[STRING]
- Here is my schema :
[image: Screen Shot 2018-10-10 at 11.21.41.png]
- Here is the JSON content sent :
[image: Screen Shot 2018-10-10 at 11.22.10.png]
Does someone has any clue ? I have been struggling 2 days with this map
data type ... :(
Thank you !
Maxime
