I tried parsing the stream without forcing it to text, like you had done Paul. I thought maybe that was the source of my problem, so I changed that to this:
def root = new JsonSlurper().setType(JsonParserType.LAX).parse(inputStream) I continue to get this error: ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error processing json fields: groovy.lang.MissingMethodException: No signature of method: Script87$_run_closure3.call() is applicable for argument types: (Node, String) values: [te=0.63847, ] Possible solutions: any(), any(), doCall(java.util.Map, java.lang.String), collect(), find(), dump() I notice the content of the error statement shows this (Node, String) values: [te=0.63847, ], which is not what I see in my input, which is this: "te": "0.9494", Is this perhaps the source of my problem - the fact that the key and value are not being read in as strings? On Sat, May 13, 2023 at 12:01 PM James McMahon <jsmcmah...@gmail.com> wrote: > Thank you Paul. I have integrated this approach into the framework of my > NiFi ExecuteScript code, which reads the flowfile content from the stream. > I am seeing an undefined method error, and it seems to indicate it cannot > digest the json. Does anything jump out at you as an obvious error in my > approach? > > Here is the simple sample json in the flowfile: > > {"id": "20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven > Process Thread-9", > > "te": "0.9494", > > "diskusage": "0.2776125422110003.3 MB", > > "memory": 77, > > "cpu": 0.58, > > "host": "172.31.73.197/ip-172-31-73-197.ec2.internal", > > "temperature": "97", > > "macaddress": "f417ead3-4fa9-4cee-a14b-7172e9ecd3ea", > > "end": "61448816405795", > "systemtime": "05/08/2023 16:52:36"} > > > Here is the output error: > > ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error processing json > fields: groovy.lang.MissingMethodException: No signature of method: > Script83$_run_closure3.call() is applicable for argument types: (Node, > String) values: [te=0.9494, ] > > Possible solutions: any(), any(), doCall(java.util.Map, java.lang.String), > collect(), find(), dump() > > > Here is the current implementation of my code: > > import groovy.json.JsonSlurper > import groovy.json.JsonParserType > import org.apache.commons.io.IOUtils > import java.nio.charset.StandardCharsets > > def keys = [] > def topValuesMap = [:].withDefault{ [:].withDefault{ 0 } } > def tallyMap = [:].withDefault{ 0 } > def tally > tally = { Map json, String prefix -> > json.each { k, v -> > String key = prefix + k > if (v instanceof List) { > tallyMap[key] += 1 > v.each{ tally(it, key + '.') } > } else { > def val = v?.toString().trim() > if (v) { > tallyMap[key] += 1 > topValuesMap[key][v] += 1 > if (v instanceof Map) tally(v, key + '.') > } > } > } > } > > > def ff = session.get() > if (!ff) return > > try { > session.read(ff, { inputStream -> > > def root = new > JsonSlurper().setType(JsonParserType.LAX).parseText(IOUtils.toString(inputStream, > StandardCharsets.UTF_8)) > > root.each { > tally(it, '') > } > } as InputStreamCallback) > > keys = tallyMap.keySet().toList() > def tallyMapString = tallyMap.collectEntries { k, v -> [(k): v] > }.toString() > def topValuesMapString = topValuesMap.collectEntries { k, v -> [(k): > v.sort{ -it.value }.take(10)] }.toString() > > ff = session.putAttribute(ff, 'triage.json.fields', keys.join(",")) > ff = session.putAttribute(ff, 'triage.json.tallyMap', tallyMapString) > ff = session.putAttribute(ff, 'triage.json.topValuesMap', > topValuesMapString) > > session.transfer(ff, REL_SUCCESS) > } catch (Exception e) { > log.error('Error processing json fields', e) > session.transfer(ff, REL_FAILURE) > } > > On Fri, May 12, 2023 at 8:54 AM Paul King <pa...@asert.com.au> wrote: > >> Something like this worked for me: >> >> def topValuesMap = [:].withDefault{ [:].withDefault{ 0 } } >> def tallyMap = [:].withDefault{ 0 } >> def tally >> tally = { Map json, String prefix -> >> json.each { k, v -> >> String key = prefix + k >> if (v instanceof List) { >> tallyMap[key] += 1 >> v.each{ tally(it, key + '.') } >> } else { >> def val = v?.toString().trim() >> if (v) { >> tallyMap[key] += 1 >> topValuesMap[key][v] += 1 >> if (v instanceof Map) tally(v, key + '.') >> } >> } >> } >> } >> >> def root = new JsonSlurper().parse(inputStream) >> root.each { // each allows json to be a list, not needed if always a map >> tally(it, '') >> } >> println tallyMap >> println topValuesMap.collectEntries{ k, m -> [(k), m.sort{ _, v -> >> -v.value }] }.take(10) >> >> >> >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> Virus-free.www.avast.com >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> <#m_1983003368347155054_m_6659555815292156536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >> On Fri, May 12, 2023 at 8:27 PM James McMahon <jsmcmah...@gmail.com> >> wrote: >> >>> Thank you for the response, Paul. I will integrate and try these >>> suggestions within my Groovy code that runs in a nifi ExecuteScript >>> processor. I'll be working on this once again tonight. >>> >>> The map topValuesMap is intended to capture this: for each key >>> identified in the json, cross-tabulate for each value associated with that >>> key how many times it occurs. After the json is fully processed for key, >>> sort the resulting map and retain only the top ten values found in the >>> json. If a set has a lastName key, the topValuesMap that results might look >>> something like this after all keys have been cross-tabulated: >>> >>> ["lastName": ["Smith" : 1023, "Jones" : 976, "Chang": 899, "Doe": 511, >>> ...], >>> "address.street": [.....], >>> . >>> . >>> . >>> "a final key": [.....] >>> ] >>> >>> Each key would have ten values in its value map, unless it >>> cross-tabulates to less than ten in total, in which case it will be sorted >>> by count value and all values accepted. >>> Again, many thanks. >>> Jim >>> >>> On Fri, May 12, 2023 at 2:19 AM Paul King <pa...@asert.com.au> wrote: >>> >>>> I am not 100% sure what you are trying to capture in topValuesMap but >>>> for tallyMap you probably want something like: >>>> >>>> def tallyMap = [:].withDefault{ 0 } >>>> def tally >>>> tally = { Map json, String prefix -> >>>> json.each { k, v -> >>>> if (v instanceof List) { >>>> tallyMap[prefix + k] += 1 >>>> v.each{ tally(it, "$prefix${k}.") } >>>> } else if (v?.toString().trim()) { >>>> tallyMap[prefix + k] += 1 >>>> if (v instanceof Map) tally(v, "$prefix${k}.") >>>> } >>>> } >>>> } >>>> >>>> def root = new JsonSlurper().parse(inputStream) >>>> def initialPrefix = '' >>>> tally(root, initialPrefix) >>>> println tallyMap >>>> >>>> Output: >>>> [name:1, age:1, address:1, address.street:1, address.city:1, >>>> address.state:1, address.zip:1, phoneNumbers:1, phoneNumbers.type:2, >>>> phoneNumbers.number:2] >>>> >>>> >>>> >>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>>> Virus-free.www.avast.com >>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>>> <#m_1983003368347155054_m_6659555815292156536_m_227840230113638997_m_3707991732434518544_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>> >>>> On Fri, May 12, 2023 at 10:38 AM James McMahon <jsmcmah...@gmail.com> >>>> wrote: >>>> >>>>> I have this incoming json: { "name": "John Doe", "age": 42, "address": >>>>> { "street": "123 Main St", "city": "Anytown", "state": "CA", "zip": >>>>> "12345" >>>>> }, "phoneNumbers": [ { "type": "home", "number": "555-1234" }, { "type": >>>>> "work", "number": "555-5678" } ] } I wish to tally all the keys in this >>>>> json in a map that gives me the key name as its key, and a count of the >>>>> number of times the key occurs in the json as its value. For this example, >>>>> the keys I expect in my output should include name, age, address, >>>>> address.street, address.city, address.state, address.zip, phoneNumbers, >>>>> phoneNumbers.type, and phoneNumbers.number. But I do not get that. >>>>> Instead, >>>>> I get this for the list of fields: triage.json.fields >>>>> name,age,address,phoneNumbers And I get this for my tally count by key: >>>>> triage.json.tallyMap [name:1, age:1, address:1, phoneNumbers:1] >>>>> >>>>> I am close, but not quite there. I don't capture all the keys. Here is >>>>> my code. How must I modify this to get the result I require? import >>>>> groovy.json.JsonSlurper import org.apache.commons.io.IOUtils import >>>>> java.nio.charset.StandardCharsets def keys = [] def tallyMap = [:] def >>>>> topValuesMap = [:] def ff = session.get() if (!ff) return try { >>>>> session.read(ff, { inputStream -> def json = new >>>>> JsonSlurper().parseText(IOUtils.toString(inputStream, >>>>> StandardCharsets.UTF_8)) json.each { k, v -> if (v != null && >>>>> !v.toString().trim().isEmpty()) { tallyMap[k] = tallyMap.containsKey(k) ? >>>>> tallyMap[k] + 1 : 1 if (topValuesMap.containsKey(k)) { def valuesMap = >>>>> topValuesMap[k] valuesMap[v] = valuesMap.containsKey(v) ? valuesMap[v] + 1 >>>>> : 1 topValuesMap[k] = valuesMap } else { topValuesMap[k] = >>>>> [:].withDefault{ >>>>> 0 }.plus([v: 1]) } } } } as InputStreamCallback) keys = >>>>> tallyMap.keySet().toList() def tallyMapString = tallyMap.collectEntries { >>>>> k, v -> [(k): v] }.toString() def topValuesMapString = >>>>> topValuesMap.collectEntries { k, v -> [(k): v.sort{ -it.value }.take(10)] >>>>> }.toString() ff = session.putAttribute(ff, 'triage.json.fields', >>>>> keys.join(",")) ff = session.putAttribute(ff, 'triage.json.tallyMap', >>>>> tallyMapString) ff = session.putAttribute(ff, 'triage.json.topValuesMap', >>>>> topValuesMapString) session.transfer(ff, REL_SUCCESS) } catch (Exception >>>>> e) >>>>> { log.error('Error processing json fields', e) session.transfer(ff, >>>>> REL_FAILURE) } >>>>> >>>>