Daeho and Matt, thank you for all your suggestions. You helped me get to a
solution. Here is how I unwound my incoming JSON with a simple flow,
My incoming JSON flowfile looks like this:
{
"KEY1":"value1",
"KEY2":"value2",
"FNAMES":["A","B","C","D"],
"KEY4":2
}
My goal is to have a flowfile for each of A, B, C, and D, with attribute
THIS_NAME set to each singular FNAMES value, and also preserving KEY1,
KEY2, KEY3 as flowfile attributes with their values pulled from the JSON.
Final flow: ListFile->FetchFile->EvaluateJsonPath->SplitJson->ExtractText
EvaluateJsonPath grabs all JSON key/values to attributes. At this point
though, FNAMES attribute is ["A","B","C","D"] -- not quite what we require.
SplitJson creates four flowfiles from one, its configuration setting
JsonPath Expression as $.FNAMES . This results in four flowfiles. We're
almost home.
The flowfile content is now just each of the singular values from FNAMES.
ExtractText creates attribute THIS_NAME configured like this:
Include Capture Group 0 false
Dynamic property added is THIS_NAME, configured to regex pattern (.*) .
(Bad idea in general in any situation where content length may vary to
large content, but not in our case where we know the values in the original
JSON list are no larger than half a KB.)
After this ExtractText step we have all our attributes, including
fragment.count of 4 and a common fragment.identifier we can later use to
reunite all after individual processing, with a MergeContent or similar.
Thank you once again.
On Thu, Dec 5, 2019 at 6:36 AM 노대호Daeho Ro <[email protected]>
wrote:
> Hm.... I might wrong.
>
> It wouldn't preserve other keys, so you have to evaluate other keys first,
> and split FNAMES and evaluate again. Sorry for the confusion.
>
> 2019년 12월 5일 (목) 오후 8:29, James McMahon <[email protected]>님이 작성:
>
>> Typo in my initial reply. I did use $.FNAMES. It drops all the other
>> key/value pairs in the output split result flowfiles.
>> I configured my SplitJSON like so:
>> JsonPathExpression $.FNAME*S*
>> Null Value Representation empty string
>>
>> If there are two values in the json array for that key FNAME*S*, I do
>> get two output flowfiles. But the only value present in the output is the
>> value from the split of the value list of FNAMES. All my other JSON keys
>> and values are not present. How do I tell SplitJSON to also retain all the
>> key/values I did not split on?
>>
>> On Thu, Dec 5, 2019 at 6:15 AM 노대호Daeho Ro <[email protected]>
>> wrote:
>>
>>> Path to be $.FNAMES, that will work I guess.
>>>
>>> 2019년 12월 5일 (목) 오후 8:10, James McMahon <[email protected]>님이 작성:
>>>
>>>> I should add that I also tried this for JsonPathExpression $.*
>>>> That result also wasn't what I require, because it gave me 14 different
>>>> flowfiles each with only one value - - the two that resulted from the FNAME
>>>> key, and one for each of the other 12 keys that had only one value.
>>>> My incoming JSON flowfile looks like this:
>>>> {
>>>> "KEY1":"value1",
>>>> "KEY2":"value2",
>>>> .
>>>> .
>>>> "FNAMES":["A","B"],
>>>> "KEY13":2
>>>> }
>>>>
>>>> This is what I need as output:
>>>> {
>>>> "KEY1":"value1",
>>>> "KEY2":"value2",
>>>> .
>>>> .
>>>> "FNAMES":"A",,
>>>> "KEY13":2
>>>> }
>>>>
>>>> and
>>>>
>>>> {
>>>> "KEY1":"value1",
>>>> "KEY2":"value2",
>>>> .
>>>> .
>>>> "FNAMES":"B",
>>>> "KEY13":2
>>>> }
>>>>
>>>> How does one configure SplitJSON to accomplish that?
>>>>
>>>> On Thu, Dec 5, 2019 at 5:59 AM James McMahon <[email protected]>
>>>> wrote:
>>>>
>>>>> Daeho, I configured my SplitJSON like so:
>>>>> JsonPathExpression $.FNAME
>>>>> Null Value Representation empty string
>>>>>
>>>>> If there are two values in the json array for that key FNAME, I do get
>>>>> two output flowfiles. But the only value present in the output is the
>>>>> value
>>>>> from the split of the list. All my other JSON keys and values are not
>>>>> present. How do I tell SplitJSON to also retain all the key/values I did
>>>>> not split on?
>>>>>
>>>>>
>>>>> On Wed, Dec 4, 2019 at 9:26 PM 노대호Daeho Ro <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Of course.
>>>>>>
>>>>>> There is a processor, the name is SplitJson. It can split the JSON
>>>>>> text by defined key. For example, if there is a key name is 'fname' and
>>>>>> has
>>>>>> the value [a, b, c]. Once you split the JSON by that processor, the
>>>>>> resulted JSON will have the same key and values for others but 'fname'
>>>>>> will
>>>>>> be a for the first JSON , b for the second and so on.
>>>>>>
>>>>>> After that, do the EvaluateJsonPath for FNAME then it will have a and
>>>>>> b and c for each splited flowfiles. Thus, I recommend you to place the
>>>>>> SplitJson processor in front of the EvaluateJsonPath processor.
>>>>>>
>>>>>> 2019년 12월 5일 (목) 오전 10:58, James McMahon <[email protected]>님이 작성:
>>>>>>
>>>>>>> I don’t quite follow, Daeho. FNAME is an attribute that results
>>>>>>> *from* EvaluateJSonPath. Can you explain what you mean by splitting
>>>>>>> the Jason key before EvaluateJSonPath?
>>>>>>> Jim
>>>>>>>
>>>>>>> On Wed, Dec 4, 2019 at 7:45 PM 노대호Daeho Ro <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I think you can split the json key for FNAME just before the
>>>>>>>> EvaluateJsonPath processor. Then, the fragment.* attributes will be
>>>>>>>> automatically created.
>>>>>>>>
>>>>>>>> 2019년 12월 5일 (목) 오전 8:24, Matt Burgess <[email protected]>님이 작성:
>>>>>>>>
>>>>>>>>> Jim,
>>>>>>>>>
>>>>>>>>> As of NiFi 1.8.0 [1], you should be able to do this with
>>>>>>>>> UpdateAttribute -> DuplicateFlowFile -> UpdateAttribute pattern,
>>>>>>>>> the
>>>>>>>>> first getting the number of values in the list via the count() EL
>>>>>>>>> function, the second using that (minus 1) to generate duplicates,
>>>>>>>>> each
>>>>>>>>> with a copy.index attribute set. That attribute can be used in
>>>>>>>>> another
>>>>>>>>> UpdateAttribute with getDelimitedField() EL function for each flow
>>>>>>>>> file to get its own value from FNAME. You may need to rename some
>>>>>>>>> of
>>>>>>>>> the attributes to fragment.* in order to use a merge processor,
>>>>>>>>> but I
>>>>>>>>> think all the necessary values are covered. Please let me know if
>>>>>>>>> this
>>>>>>>>> works for you or not, I added various improvements in order to
>>>>>>>>> support
>>>>>>>>> use cases like this, but if I missed something I can certainly add
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/NIFI-5454
>>>>>>>>>
>>>>>>>>> On Wed, Dec 4, 2019 at 4:54 PM James McMahon <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > I have a series of attributes that result from an
>>>>>>>>> EvaluateJSonPath. One of those attributes, FNAME, appears to be a
>>>>>>>>> list of
>>>>>>>>> values like so: [“A”,”B”,”C”]. I want to split my flow file into one
>>>>>>>>> for
>>>>>>>>> each list element. I need my results to have the original content,
>>>>>>>>> all the
>>>>>>>>> original attributes, and its value for the split result out of the
>>>>>>>>> list as
>>>>>>>>> a new attribute. I need to also know the split count, and be able to
>>>>>>>>> later
>>>>>>>>> merge my flow files after evaluating the results of the split.
>>>>>>>>> > How can I accomplish this?
>>>>>>>>> > Thanks very much in advance.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> 노대호 *Daeho Ro */ Service Dev.
>>>>>>>> [email protected] / *M* +82 10-6366-2636
>>>>>>>> *KR* 06167 서울시 서초구 강남대로 327, 13층
>>>>>>>> <https://www.google.com/maps/search/%EC%84%9C%EC%9A%B8%EC%8B%9C+%EC%84%9C%EC%B4%88%EA%B5%AC+%EA%B0%95%EB%82%A8%EB%8C%80%EB%A1%9C+327,+13%EC%B8%B5?entry=gmail&source=g>
>>>>>>>> [image: Bespin Global] <https://bespinglobal.com/>
>>>>>>>> 국내 최다 클라우드 인증 자격을 보유한 MSP • 국내 유일 ISO 인증을
>>>>>>>> 확보한 MSP • 가트너가 인정한 한중일 유일한 MSP
>>>>>>>> www.bespinglobal.com • ISO 27001:2013 • ISO 9001:2015 Certified
>>>>>>>> ------------------------------
>>>>>>>> *Confidentiality Note:* This email may contain confidential and/or
>>>>>>>> private information.
>>>>>>>> If you received this email in error please delete and notify the
>>>>>>>> sender.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> 노대호 *Daeho Ro */ Service Dev.
>>>>>> [email protected] / *M* +82 10-6366-2636
>>>>>> *KR* 06167 서울시 서초구 강남대로 327, 13층
>>>>>> [image: Bespin Global] <https://bespinglobal.com/>
>>>>>> 국내 최다 클라우드 인증 자격을 보유한 MSP • 국내 유일 ISO 인증을
>>>>>> 확보한 MSP • 가트너가 인정한 한중일 유일한 MSP
>>>>>> www.bespinglobal.com • ISO 27001:2013 • ISO 9001:2015 Certified
>>>>>> ------------------------------
>>>>>> *Confidentiality Note:* This email may contain confidential and/or
>>>>>> private information.
>>>>>> If you received this email in error please delete and notify the
>>>>>> sender.
>>>>>>
>>>>>
>>>
>>> --
>>> 노대호 *Daeho Ro */ Service Dev.
>>> [email protected] / *M* +82 10-6366-2636
>>> *KR* 06167 서울시 서초구 강남대로 327, 13층
>>> [image: Bespin Global] <https://bespinglobal.com/>
>>> 국내 최다 클라우드 인증 자격을 보유한 MSP • 국내 유일 ISO 인증을
>>> 확보한 MSP • 가트너가 인정한 한중일 유일한 MSP
>>> www.bespinglobal.com • ISO 27001:2013 • ISO 9001:2015 Certified
>>> ------------------------------
>>> *Confidentiality Note:* This email may contain confidential and/or
>>> private information.
>>> If you received this email in error please delete and notify the sender.
>>>
>>
>
> --
> 노대호 *Daeho Ro */ Service Dev.
> [email protected] / *M* +82 10-6366-2636
> *KR* 06167 서울시 서초구 강남대로 327, 13층
> [image: Bespin Global] <https://bespinglobal.com/>
> 국내 최다 클라우드 인증 자격을 보유한 MSP • 국내 유일 ISO 인증을
> 확보한 MSP • 가트너가 인정한 한중일 유일한 MSP
> www.bespinglobal.com • ISO 27001:2013 • ISO 9001:2015 Certified
> ------------------------------
> *Confidentiality Note:* This email may contain confidential and/or
> private information.
> If you received this email in error please delete and notify the sender.
>
