Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-19 Thread via GitHub 


mattyb149 closed pull request #7746: NIFI-8135 allow CHOICE data types in 
conversion of Records to Java Maps
URL: https://github.com/apache/nifi/pull/7746


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-19 Thread via GitHub 


mattyb149 commented on PR #7746:
URL: https://github.com/apache/nifi/pull/7746#issuecomment-1771402603

   +1 LGTM, tested various scenarios and got expected results. Thanks for the 
fix! Merging to support/nifi-1.x and main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-17 Thread via GitHub 


ChrisSamo632 commented on PR #7746:
URL: https://github.com/apache/nifi/pull/7746#issuecomment-1765802460

   @mattyb149 certainly 2, 1 as well if it doesn't break things (I think it 
should be OK). Jira ticket updated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-17 Thread via GitHub 


mattyb149 commented on PR #7746:
URL: https://github.com/apache/nifi/pull/7746#issuecomment-1765772261

   Is this meant for both 1.x and 2.x? Can you update the Jira with `1.latest` 
and/or `2.latest` to indicate where you'd like to see this merged? Please and 
thanks! I'll give it a try on main in the meantime :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-11 Thread via GitHub 


ChrisSamo632 commented on code in PR #7746:
URL: https://github.com/apache/nifi/pull/7746#discussion_r1355749268


##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -854,79 +855,82 @@ public static Map toMap(final Object 
value, final String fieldNa
  */
 @SuppressWarnings({"unchecked", "rawtypes"})
 public static Object convertRecordFieldtoObject(final Object value, final 
DataType dataType) {
-
 if (value == null) {
 return null;
 }
 
+DataType chosenDataType;
+if (dataType instanceof ChoiceDataType) {
+final DataType chosen = chooseDataType(value, (ChoiceDataType) 
dataType);
+chosenDataType = chosen != null ? chosen : dataType;
+} else {
+chosenDataType = dataType;
+}
+
 if (value instanceof Record) {
-Record record = (Record) value;
-RecordSchema recordSchema = record.getSchema();
+final Record record = (Record) value;
+final RecordSchema recordSchema = record.getSchema();
 if (recordSchema == null) {
 throw new IllegalTypeConversionException("Cannot convert value 
of type Record to Map because Record does not have an associated Schema");
 }
 
-final Map recordMap = new LinkedHashMap<>();
-for (RecordField field : recordSchema.getFields()) {
-final DataType fieldDataType = field.getDataType();
+final Map recordMap = new 
LinkedHashMap<>(record.getRawFieldNames().size(), 1);
+for (final RecordField field : recordSchema.getFields()) {
+if (field.getDataType() instanceof ChoiceDataType) {
+final DataType chosen = chooseDataType(value, 
(ChoiceDataType) field.getDataType());
+chosenDataType = chosen != null ? chosen : 
field.getDataType();
+} else {
+chosenDataType = field.getDataType();
+}
 final String fieldName = field.getFieldName();
-Object fieldValue = record.getValue(fieldName);
+final Object fieldValue = record.getValue(fieldName);
 
 if (fieldValue == null) {
 recordMap.put(fieldName, null);
-} else if (isScalarValue(fieldDataType, fieldValue)) {
+} else if (isScalarValue(chosenDataType, fieldValue)) {
 recordMap.put(fieldName, fieldValue);
-} else if (fieldDataType instanceof RecordDataType) {
+} else if (chosenDataType instanceof RecordDataType) {
 Record nestedRecord = (Record) fieldValue;

Review Comment:
   Resolved, the `chooseDataType` call needed to use the `fieldValue`, not the 
`value` (of the Record passed to the method)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-8135 allow CHOICE data types in conversion of Records to Java Maps [nifi]

2023-10-11 Thread via GitHub 


ChrisSamo632 commented on code in PR #7746:
URL: https://github.com/apache/nifi/pull/7746#discussion_r1354608522


##
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java:
##
@@ -854,79 +855,82 @@ public static Map toMap(final Object 
value, final String fieldNa
  */
 @SuppressWarnings({"unchecked", "rawtypes"})
 public static Object convertRecordFieldtoObject(final Object value, final 
DataType dataType) {
-
 if (value == null) {
 return null;
 }
 
+DataType chosenDataType;
+if (dataType instanceof ChoiceDataType) {
+final DataType chosen = chooseDataType(value, (ChoiceDataType) 
dataType);
+chosenDataType = chosen != null ? chosen : dataType;
+} else {
+chosenDataType = dataType;
+}
+
 if (value instanceof Record) {
-Record record = (Record) value;
-RecordSchema recordSchema = record.getSchema();
+final Record record = (Record) value;
+final RecordSchema recordSchema = record.getSchema();
 if (recordSchema == null) {
 throw new IllegalTypeConversionException("Cannot convert value 
of type Record to Map because Record does not have an associated Schema");
 }
 
-final Map recordMap = new LinkedHashMap<>();
-for (RecordField field : recordSchema.getFields()) {
-final DataType fieldDataType = field.getDataType();
+final Map recordMap = new 
LinkedHashMap<>(record.getRawFieldNames().size(), 1);
+for (final RecordField field : recordSchema.getFields()) {
+if (field.getDataType() instanceof ChoiceDataType) {
+final DataType chosen = chooseDataType(value, 
(ChoiceDataType) field.getDataType());
+chosenDataType = chosen != null ? chosen : 
field.getDataType();
+} else {
+chosenDataType = field.getDataType();
+}
 final String fieldName = field.getFieldName();
-Object fieldValue = record.getValue(fieldName);
+final Object fieldValue = record.getValue(fieldName);
 
 if (fieldValue == null) {
 recordMap.put(fieldName, null);
-} else if (isScalarValue(fieldDataType, fieldValue)) {
+} else if (isScalarValue(chosenDataType, fieldValue)) {
 recordMap.put(fieldName, fieldValue);
-} else if (fieldDataType instanceof RecordDataType) {
+} else if (chosenDataType instanceof RecordDataType) {
 Record nestedRecord = (Record) fieldValue;

Review Comment:
   Recent example from the [NiFi Dev Mailing 
List](https://lists.apache.org/thread/kcnsxvbbdfwfhj0tdsyn53x8ljhgdt1v) 
containing some failing data and associated Jolt Transform Spec.
   
   Currently fails on this branch with:
   ```
   java.lang.ClassCastException: class java.lang.String cannot be cast to class 
org.apache.nifi.serialization.record.Record (java.lang.String is in module 
java.base of loader 'bootstrap'; org.apache.nifi.serialization.record.Record is 
in unnamed module of loader org.apache.nifi.nar.NarClassLoader @4b3ad7ca)
at 
org.apache.nifi.serialization.record.util.DataTypeUtils.convertRecordFieldtoObject(DataTypeUtils.java:893)
at 
org.apache.nifi.processors.jolt.record.JoltTransformRecord.transform(JoltTransformRecord.java:425)
   ```
   
   Fails on `main`/`support/nifi-1.x` with:
   
   ```
   Cannot convert value [MapRecord[{A: 192,B: 32,C: 13,D: 19}]] of
   CHOICE[STRING,RECORD] to Map for field dimensions because type is not
   supported.
   ```
   
   So this change for NIFI-8135 has at least had an impact, albeit not quite 
resolved the problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org