Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 merged PR #7745: URL: https://github.com/apache/nifi/pull/7745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-249151 Thanks for the update @ChrisSamo632 all looks good to me at this point. Sorry this has taken so long, and thanks for sticking with it! +1 will merge to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
joewitt commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-2094286612 Thanks for rebasing @ChrisSamo632 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-2094055528 Rebased against latest `main` to refactor changes into new `nifi-extension-bundles` module -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1969913218 @markap14 thanks for the review, I've addressed your comments (and reverted the IDE auto-formatting issues!) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1496517940
##
nifi-docs/src/main/asciidoc/record-path-guide.adoc:
##
@@ -945,6 +949,27 @@ The following record path expression would return:
| `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map)
|==
+*Note* that Maps cannot be serialised with a RecordSetWriter.
+
+Given a record such as:
+
+
+{
+ "json_str": "{\"name\":\"John\",\"age\":30,\"addresses\"[{\"address_1\":
\"123 Fake Street\"}]}"
+}
+
+
+The following record path expression would return:
+
+|==
+| RecordPath | Return value
+| `unescapeJson(/json_str, 'false')` | {"name"="John", "age"=30,
"addresses"=[{address_1"="123 Fake Street"}]} (as a Map, with each entry in
"addresses" as a Map)
+| `unescapeJson(/json_str, 'true', 'false')` | {"name": "John", "age": 30,
"addresses"=[{address_1"="123 Fake Street"}]} (as a Record, with each entry in
"addresses" as a Map)
+| `unescapeJson(/json_str, 'true', 'true')` | {"name": "John", "age": 30,
"addresses": [{"address_1": "123 Fake Street"}]} (as a Record, with each entry
in "addresses" as a Record)
+|==
+
+*Note* that Maps cannot be serialised with a RecordSetWriter.
Review Comment:
This is the third time that this is stated in the documentation for
`unescapeJson`. I think stating it once is enough.
##
nifi-docs/src/main/asciidoc/record-path-guide.adoc:
##
@@ -894,7 +894,11 @@ The following record path expression would convert the
record into an escaped JS
=== unescapeJson
Converts a stringified JSON element to a Record, Array or simple field (e.g.
String), using the UTF-8 character set.
-Optionally convert JSON Objects parsed as Maps into Records (defaults to
false).
+Optionally convert JSON Objects parsed as Maps into Records (defaults to
false), recursively (defaults to false).
Review Comment:
Can you clarify this statement? I.e., explicitly state the arguments that
the function takes, their names, and whether or not they are required?
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestUpdateRecord.java:
##
@@ -232,16 +235,20 @@ public void testChangingSchema() throws
InitializationException, IOException {
final JsonTreeReader jsonReader = new JsonTreeReader();
runner.addControllerService("reader", jsonReader);
-final String inputSchemaText = new
String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc")));
-final String outputSchemaText = new
String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-string.avsc")));
+final String inputSchemaText = new String(Files
+
.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc")));
+final String outputSchemaText = new String(Files
+
.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-string.avsc")));
-runner.setProperty(jsonReader,
SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY,
SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
+runner.setProperty(jsonReader,
SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY,
+SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT,
inputSchemaText);
runner.enableControllerService(jsonReader);
final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
runner.addControllerService("writer", jsonWriter);
-runner.setProperty(jsonWriter,
SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY,
SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
+runner.setProperty(jsonWriter,
SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY,
+SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
Review Comment:
There are a huge number of edits to this file, but almost all of them appear
to be inserting arbitrary newlines that are not necessary. Can we fix that?
##
nifi-docs/src/main/asciidoc/record-path-guide.adoc:
##
@@ -945,6 +949,27 @@ The following record path expression would return:
| `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map)
|==
+*Note* that Maps cannot be serialised with a RecordSetWriter.
Review Comment:
This seems to be a restatement of the above comment (though the above
comment seems more clear to me). Does it make sense to remove this?
##
nifi-docs/src/main/asciidoc/record-path-guide.adoc:
##
@@ -945,6 +949,27 @@ The following record path expression would return:
| `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map)
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1954700969 Thanks @ChrisSamo632 will take a look this week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1949284657 Have rebased from latest `main` to address merge conflicts, this PR should be ready for re-review @markap14 / @exceptionfactory -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420401479
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S
final RecordField recordField = new RecordField(key,
inferredDataType, true);
inferredFieldTypes.add(recordField);
-final Object coercedValue = convertType(rawValue,
inferredDataType, fieldName, charset);
-coercedValues.put(key, coercedValue);
+convertType(rawValue, inferredDataType, fieldName, charset);
}
-final RecordSchema inferredSchema = new
SimpleRecordSchema(inferredFieldTypes);
-return inferredSchema;
+return new SimpleRecordSchema(inferredFieldTypes);
}
public static Record toRecord(final Object value, final String fieldName,
final Charset charset) {
+return toRecord(value, fieldName, charset, false);
+}
+
+private static Object covertObjectToRecord(final Object rawValue, final
String key, final Charset charset) {
+final Object coercedValue;
+if (rawValue instanceof Map) {
+coercedValue = toRecord(rawValue, key, charset, true);
+} else if (rawValue instanceof Object[]) {
+final Object[] objArray = (Object[]) rawValue;
+coercedValue = Arrays.stream(objArray).noneMatch(o -> o instanceof
Map)
+? objArray
+: Arrays.stream(objArray).map(o -> toRecord(o, key,
charset, true)).toArray();
+} else if (rawValue instanceof Collection) {
+final Collection objCollection = (Collection) rawValue;
+// Records have ARRAY DataTypes, so convert any Collections
+coercedValue = objCollection.stream().noneMatch(o -> o instanceof
Map)
+? objCollection.toArray()
+: objCollection.stream().map(o -> toRecord(o, key,
charset, true)).toArray();
Review Comment:
That's fair, I've reverted this and found a couple of other Streams within
`DataTypeUtils` that should probably be changed too, will include in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420713574
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
Absolutely. The only problem is that `Map.of` doesn't provide deterministic
ordering, so we do need to use `LinkedHashMap`s in some places, which I've
changed to use the approach suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1878261969 @markap14 grateful if you've time to re-review this change at some point -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420713574
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
Absolutely. The only problem is that `Map.of` doesn't provide deterministic
ordering, so we do need to use `LinkedHasMap`s in some places, which I've
changed to use the approach suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
exceptionfactory commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420465749
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
Yes, Map.of() is much better in many ways, and it is unfortunate that some
portions of test code use the initialization approach out of convenience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420434506
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
I've never been a particular fan of the `{{ ... }}` constructor approach,
but I've seen it throughout lots of NiFi modules, so had just been copying the
style.
With the advent of `Map.of`, that's a better approach anyway I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420402511
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
Review Comment:
I'd been sticking with Java 8 compatible code to get this onto the
`support/1.x` branch, but if we're happy with it being only NiFi 2.x now, I'll
update these tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1420401479
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S
final RecordField recordField = new RecordField(key,
inferredDataType, true);
inferredFieldTypes.add(recordField);
-final Object coercedValue = convertType(rawValue,
inferredDataType, fieldName, charset);
-coercedValues.put(key, coercedValue);
+convertType(rawValue, inferredDataType, fieldName, charset);
}
-final RecordSchema inferredSchema = new
SimpleRecordSchema(inferredFieldTypes);
-return inferredSchema;
+return new SimpleRecordSchema(inferredFieldTypes);
}
public static Record toRecord(final Object value, final String fieldName,
final Charset charset) {
+return toRecord(value, fieldName, charset, false);
+}
+
+private static Object covertObjectToRecord(final Object rawValue, final
String key, final Charset charset) {
+final Object coercedValue;
+if (rawValue instanceof Map) {
+coercedValue = toRecord(rawValue, key, charset, true);
+} else if (rawValue instanceof Object[]) {
+final Object[] objArray = (Object[]) rawValue;
+coercedValue = Arrays.stream(objArray).noneMatch(o -> o instanceof
Map)
+? objArray
+: Arrays.stream(objArray).map(o -> toRecord(o, key,
charset, true)).toArray();
+} else if (rawValue instanceof Collection) {
+final Collection objCollection = (Collection) rawValue;
+// Records have ARRAY DataTypes, so convert any Collections
+coercedValue = objCollection.stream().noneMatch(o -> o instanceof
Map)
+? objCollection.toArray()
+: objCollection.stream().map(o -> toRecord(o, key,
charset, true)).toArray();
Review Comment:
That's fair, I've reverted this and found a coupleof other Streams within
`DataTypeUtils` that should probably be changed too, will include in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on code in PR #7745:
URL: https://github.com/apache/nifi/pull/7745#discussion_r1417519143
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
+);
+// recursively convert Maps to Records (addresses becomes and ARRAY or
RECORDs)
+assertEquals(
+DataTypeUtils.toRecord(new LinkedHashMap(){{
+put("firstName", "John");
+put("age", 30);
+put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
+}}, "json_str"),
Review Comment:
We should be creating subclasses of LinkedHashMap for initialization
purposes. Should just declare a LinkedHashMap and add elements to it:
```
Map values = new LinkedHashMap<>();
values.put("firstName", "John");
values.put("age", 30);
values.put("addresses", new Object[]
{DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake
Street"), "addresses")});
```
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue()
+);
+
+// test nested Record converted from Map Object
+final Record nestedRecordFromMap = new MapRecord(schema,
+Collections.singletonMap(
+"json_str",
+
"{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake
Street\"}]}")
Review Comment:
Would probably be cleaner here to use a text block here:
```
"""
{"firstName":"John","age":30,"address"...}"""
```
##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S
final RecordField recordField = new RecordField(key,
inferredDataType, true);
inferredFieldTypes.add(recordField);
-final Object coercedValue = convertType(rawValue,
inferredDataType, fieldName, charset);
-coercedValues.put(key, coercedValue);
+convertType(rawValue, inferredDataType, fieldName, charset);
}
-final RecordSchema inferredSchema = new
SimpleRecordSchema(inferredFieldTypes);
-return inferredSchema;
+return new SimpleRecordSchema(inferredFieldTypes);
}
public static Record toRecord(final Object value, final String fieldName,
final Charset charset) {
+return toRecord(value, fieldName, charset, false);
+}
+
+private static Object covertObjectToRecord(final Object rawValue, final
String key, final Charset charset) {
Review Comment:
typo - name is `covert...` instead of `convert...`
But we should not have a method named `convertObjectToRecord` that returns
an Object that may be an array, may be a Record, or may be any other object
that was provided. Need to either always return a Record or create a method
name that reflects what is truly done.
##
nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java:
##
@@ -1837,7 +1838,41 @@ public void testUnescapeJson() {
put("firstName", "John");
put("age", 30);
}}, "json_str"),
-RecordPath.compile("unescapeJson(/json_str,
'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue()
+
