Thank you Paul. I have integrated this approach into the framework of my
NiFi ExecuteScript code, which reads the flowfile content from the stream.
I am seeing an undefined method error, and it seems to indicate it cannot
digest the json. Does anything jump out at you as an obvious error in my
approach?
Here is the simple sample json in the flowfile:
{"id": "20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven
Process Thread-9",
"te": "0.9494",
"diskusage": "0.2776125422110003.3 MB",
"memory": 77,
"cpu": 0.58,
"host": "172.31.73.197/ip-172-31-73-197.ec2.internal",
"temperature": "97",
"macaddress": "f417ead3-4fa9-4cee-a14b-7172e9ecd3ea",
"end": "61448816405795",
"systemtime": "05/08/2023 16:52:36"}
Here is the output error:
ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error
processing json fields: groovy.lang.MissingMethodException: No
signature of method: Script83$_run_closure3.call() is applicable for
argument types: (Node, String) values: [te=0.9494, ]
Possible solutions: any(), any(), doCall(java.util.Map,
java.lang.String), collect(), find(), dump()
Here is the current implementation of my code:
import groovy.json.JsonSlurper
import groovy.json.JsonParserType
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
def keys = []
def topValuesMap = [:].withDefault{ [:].withDefault{ 0 } }
def tallyMap = [:].withDefault{ 0 }
def tally
tally = { Map json, String prefix ->
json.each { k, v ->
String key = prefix + k
if (v instanceof List) {
tallyMap[key] += 1
v.each{ tally(it, key + '.') }
} else {
def val = v?.toString().trim()
if (v) {
tallyMap[key] += 1
topValuesMap[key][v] += 1
if (v instanceof Map) tally(v, key + '.')
}
}
}
}
def ff = session.get()
if (!ff) return
try {
session.read(ff, { inputStream ->
def root = new
JsonSlurper().setType(JsonParserType.LAX).parseText(IOUtils.toString(inputStream,
StandardCharsets.UTF_8))
root.each {
tally(it, '')
}
} as InputStreamCallback)
keys = tallyMap.keySet().toList()
def tallyMapString = tallyMap.collectEntries { k, v -> [(k): v]
}.toString()
def topValuesMapString = topValuesMap.collectEntries { k, v -> [(k):
v.sort{ -it.value }.take(10)] }.toString()
ff = session.putAttribute(ff, 'triage.json.fields', keys.join(","))
ff = session.putAttribute(ff, 'triage.json.tallyMap', tallyMapString)
ff = session.putAttribute(ff, 'triage.json.topValuesMap',
topValuesMapString)
session.transfer(ff, REL_SUCCESS)
} catch (Exception e) {
log.error('Error processing json fields', e)
session.transfer(ff, REL_FAILURE)
}
On Fri, May 12, 2023 at 8:54 AM Paul King <pa...@asert.com.au> wrote:
> Something like this worked for me:
>
> def topValuesMap = [:].withDefault{ [:].withDefault{ 0 } }
> def tallyMap = [:].withDefault{ 0 }
> def tally
> tally = { Map json, String prefix ->
> json.each { k, v ->
> String key = prefix + k
> if (v instanceof List) {
> tallyMap[key] += 1
> v.each{ tally(it, key + '.') }
> } else {
> def val = v?.toString().trim()
> if (v) {
> tallyMap[key] += 1
> topValuesMap[key][v] += 1
> if (v instanceof Map) tally(v, key + '.')
> }
> }
> }
> }
>
> def root = new JsonSlurper().parse(inputStream)
> root.each { // each allows json to be a list, not needed if always a map
> tally(it, '')
> }
> println tallyMap
> println topValuesMap.collectEntries{ k, m -> [(k), m.sort{ _, v ->
> -v.value }] }.take(10)
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_6659555815292156536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Fri, May 12, 2023 at 8:27 PM James McMahon <jsmcmah...@gmail.com>
> wrote:
>
>> Thank you for the response, Paul. I will integrate and try these
>> suggestions within my Groovy code that runs in a nifi ExecuteScript
>> processor. I'll be working on this once again tonight.
>>
>> The map topValuesMap is intended to capture this: for each key identified
>> in the json, cross-tabulate for each value associated with that key how
>> many times it occurs. After the json is fully processed for key, sort the
>> resulting map and retain only the top ten values found in the json. If a
>> set has a lastName key, the topValuesMap that results might look something
>> like this after all keys have been cross-tabulated:
>>
>> ["lastName": ["Smith" : 1023, "Jones" : 976, "Chang": 899, "Doe": 511,
>> ...],
>> "address.street": [.....],
>> .
>> .
>> .
>> "a final key": [.....]
>> ]
>>
>> Each key would have ten values in its value map, unless it
>> cross-tabulates to less than ten in total, in which case it will be sorted
>> by count value and all values accepted.
>> Again, many thanks.
>> Jim
>>
>> On Fri, May 12, 2023 at 2:19 AM Paul King <pa...@asert.com.au> wrote:
>>
>>> I am not 100% sure what you are trying to capture in topValuesMap but
>>> for tallyMap you probably want something like:
>>>
>>> def tallyMap = [:].withDefault{ 0 }
>>> def tally
>>> tally = { Map json, String prefix ->
>>> json.each { k, v ->
>>> if (v instanceof List) {
>>> tallyMap[prefix + k] += 1
>>> v.each{ tally(it, "$prefix${k}.") }
>>> } else if (v?.toString().trim()) {
>>> tallyMap[prefix + k] += 1
>>> if (v instanceof Map) tally(v, "$prefix${k}.")
>>> }
>>> }
>>> }
>>>
>>> def root = new JsonSlurper().parse(inputStream)
>>> def initialPrefix = ''
>>> tally(root, initialPrefix)
>>> println tallyMap
>>>
>>> Output:
>>> [name:1, age:1, address:1, address.street:1, address.city:1,
>>> address.state:1, address.zip:1, phoneNumbers:1, phoneNumbers.type:2,
>>> phoneNumbers.number:2]
>>>
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> Virus-free.www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_6659555815292156536_m_227840230113638997_m_3707991732434518544_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>> On Fri, May 12, 2023 at 10:38 AM James McMahon <jsmcmah...@gmail.com>
>>> wrote:
>>>
>>>> I have this incoming json: { "name": "John Doe", "age": 42, "address":
>>>> { "street": "123 Main St", "city": "Anytown", "state": "CA", "zip": "12345"
>>>> }, "phoneNumbers": [ { "type": "home", "number": "555-1234" }, { "type":
>>>> "work", "number": "555-5678" } ] } I wish to tally all the keys in this
>>>> json in a map that gives me the key name as its key, and a count of the
>>>> number of times the key occurs in the json as its value. For this example,
>>>> the keys I expect in my output should include name, age, address,
>>>> address.street, address.city, address.state, address.zip, phoneNumbers,
>>>> phoneNumbers.type, and phoneNumbers.number. But I do not get that. Instead,
>>>> I get this for the list of fields: triage.json.fields
>>>> name,age,address,phoneNumbers And I get this for my tally count by key:
>>>> triage.json.tallyMap [name:1, age:1, address:1, phoneNumbers:1]
>>>>
>>>> I am close, but not quite there. I don't capture all the keys. Here is
>>>> my code. How must I modify this to get the result I require? import
>>>> groovy.json.JsonSlurper import org.apache.commons.io.IOUtils import
>>>> java.nio.charset.StandardCharsets def keys = [] def tallyMap = [:] def
>>>> topValuesMap = [:] def ff = session.get() if (!ff) return try {
>>>> session.read(ff, { inputStream -> def json = new
>>>> JsonSlurper().parseText(IOUtils.toString(inputStream,
>>>> StandardCharsets.UTF_8)) json.each { k, v -> if (v != null &&
>>>> !v.toString().trim().isEmpty()) { tallyMap[k] = tallyMap.containsKey(k) ?
>>>> tallyMap[k] + 1 : 1 if (topValuesMap.containsKey(k)) { def valuesMap =
>>>> topValuesMap[k] valuesMap[v] = valuesMap.containsKey(v) ? valuesMap[v] + 1
>>>> : 1 topValuesMap[k] = valuesMap } else { topValuesMap[k] = [:].withDefault{
>>>> 0 }.plus([v: 1]) } } } } as InputStreamCallback) keys =
>>>> tallyMap.keySet().toList() def tallyMapString = tallyMap.collectEntries {
>>>> k, v -> [(k): v] }.toString() def topValuesMapString =
>>>> topValuesMap.collectEntries { k, v -> [(k): v.sort{ -it.value }.take(10)]
>>>> }.toString() ff = session.putAttribute(ff, 'triage.json.fields',
>>>> keys.join(",")) ff = session.putAttribute(ff, 'triage.json.tallyMap',
>>>> tallyMapString) ff = session.putAttribute(ff, 'triage.json.topValuesMap',
>>>> topValuesMapString) session.transfer(ff, REL_SUCCESS) } catch (Exception e)
>>>> { log.error('Error processing json fields', e) session.transfer(ff,
>>>> REL_FAILURE) }
>>>>
>>>
