[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782454#comment-16782454
]
Joseph Percivall commented on NIFI-5678:
[~joewitt] yeah I debated about creating a new one but figured it may be that
the issue wasn't fixed and may need to be reopened. I'll go ahead and create a
new one.
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781313#comment-16781313
]
Joseph Witt commented on NIFI-5678:
---
[~JPercivall] you'll want to create a new JIRA with the details specified.
comments on a resolved JIRA often go silent...
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781299#comment-16781299
]
Joseph Percivall commented on NIFI-5678:
[~mattyb149] I just attempted to do this with a v1.9.0 instance and it hits the
same error message ("Value is of type
org.apache.nifi.serialization.record.MapRecord but was expected to be of type
MAP"). Attached is a template to recreate it. A potential reason is
that the map is nested within an array of records.
[^Nested_map_record_failing_validation.xml]
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650593#comment-16650593
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3060
Thanks @mattyb149 all looks good to me, was able to verify that the unit
test performs as expected and manual testing (including the example from the
dev@ mailing list, where I think this JIRA originated from) passes my tests. +1
merged to master.
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650592#comment-16650592
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3060
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650590#comment-16650590
]
ASF subversion and git services commented on NIFI-5678:
---
Commit 218063a0b57c609cbc30276907c0c254d88c699c in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=218063a ]
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
This closes #3060.
Signed-off-by: Mark Payne
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645442#comment-16645442
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224208847
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Sorry, I looked through the code and understand. excuse me
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645433#comment-16645433
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3060#discussion_r224205938
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java
---
@@ -196,21 +196,32 @@ private boolean isTypeCorrect(final Object value,
final DataType dataType) {
return true;
case MAP:
-if (!(value instanceof Map)) {
-return false;
-}
-
-final MapDataType mapDataType = (MapDataType) dataType;
-final DataType valueDataType = mapDataType.getValueType();
-final Map map = (Map) value;
-
-for (final Object mapValue : map.values()) {
-if (!isTypeCorrect(mapValue, valueDataType)) {
-return false;
+if (value instanceof Map) {
+final MapDataType mapDataType = (MapDataType) dataType;
+final DataType valueDataType =
mapDataType.getValueType();
+final Map map = (Map) value;
+
+for (final Object mapValue : map.values()) {
+if (!isTypeCorrect(mapValue, valueDataType)) {
+return false;
+}
}
+return true;
--- End diff --
Would this be the same if it was a MapRecord?
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645423#comment-16645423
]
ASF GitHub Bot commented on NIFI-5678:
--
GitHub user mattyb149 opened a pull request:
https://github.com/apache/nifi/pull/3060
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mattyb149/nifi NIFI-5678
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3060.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3060
commit c66de368d07f37a8652c09e4b116b232972821ac
Author: Matthew Burgess
Date: 2018-10-10T19:01:40Z
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645129#comment-16645129
]
Matt Burgess commented on NIFI-5678:
ValidateRecord checks only for the field object being a Map. Additionally it
should support MapRecord, as this is how objects often are exposed by the
Record API (whether they are of Map or Record type).
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
