RE: attributesToJSON order

[Paul Gibeault (pagibeault)]

Bryan and Aldrin,
  Thanks for your replies.

The user has already specified an order in the attributesToJSON processor’s 
configuration.  If we honor this order, we should be covered.  Changing the 
datatype to LinkedHashMap is exactly what I had in mind.  I do not believe we 
need to worry about impacting legacy behavior here as users are already 
accustomed to not depending on the order of the JSON values.

This became an issue for us when, downstream, we used a custom processor 
JSONToDelimited to export the JSON content as a CSV string.  We were unable to 
trust the order of the JSON elements which resulted in invalid ordering of the 
values in our CSV.

We have plans to provide additional configuration on our JSONToDelimited 
processor to specify which JSON keys to export and in which order.  In the 
meantime, we are using the replaceText processor with the attributes called out 
explicitly.  This is messy but effective.

Thanks,
Paul Gibeault

From: Bryan Rosander [mailto:bryanrosan...@gmail.com]
Sent: Thursday, October 20, 2016 1:43 PM
To: users@nifi.apache.org
Cc: Peter Wicks (pwicks) <pwi...@micron.com>
Subject: Re: attributesToJSON order

I'm not sure of the use case either but if there is one, it should be pretty 
easy to let the user select the type of Map they'd prefer to store it in.

HashMap should probably be default if order doesn't matter, LinkedHashMap if 
you want to maintain insertion order, TreeMap if you want alphabetical order.

Thanks,
Bryan

On Thu, Oct 20, 2016 at 3:27 PM, Aldrin Piri 
<aldrinp...@gmail.com<mailto:aldrinp...@gmail.com>> wrote:
Hey Paul,

Could you highlight the use case you are looking to address or shortcoming that 
has emerged because of this?  No strong qualms with providing it, just not sure 
I am tracking where this becomes problematic.

Thanks,
Aldrin

On Thu, Oct 20, 2016 at 3:23 PM, Paul Gibeault (pagibeault) 
<pagibea...@micron.com<mailto:pagibea...@micron.com>> wrote:
Hello all,
  I thought I would run this by you before I created a Jira ticket.

The processor attributesToJSON does not create a JSON document with key/values 
in the same order as provided in the processor’s configuration.

Example:
  AttributesList: computationName,computationType,strategyName

Output:
{
  " strategyName" : "blue",
  " computationType" : "21DC8X32",
  " computationName" : "453d6c4f-fdd-e611-80c9-0050233e88"
}

This behavior is coming from the datatype used in the attributesToJSON 
processor:

    protected Map<String, String> buildAttributesMapForFlowFile(FlowFile ff, 
String atrList,
                                                                boolean 
includeCoreAttributes,
                                                                boolean 
nullValForEmptyString) {

        Map<String, String> atsToWrite = new HashMap<>();

        . . .
    }

Using another datatype that preserved order would correct this behavior.  The 
JSON specification does mention that the object list is order independent.  
This does not necessarily mean we should cause the disorder though.

Should we create a JIRA ticket and solution for this?

Thanks,
Paul Gibeault