Re: NiFi ValidateRecord - unable to handle missing mandatory ARRAY ?

2019-12-05 Thread Emanuel Oliveira
Hi Mark, be sure you copy paste "NOK - payload BAD 1 - " into GenerateFlowfile as this is the problem. Cheers, Emanuel On Thu 5 Dec 2019, 22:03 Mark Payne, wrote: > Emanuel, > > What version of NiFi are you using? > > I just tested the attached template against the latest, and the FlowFile >

Re: NiFi ValidateRecord - unable to handle missing mandatory ARRAY ?

2019-12-05 Thread Mark Payne
Emanuel, What version of NiFi are you using? I just tested the attached template against the latest, and the FlowFile was routed to 'invalid' with the explanation: Records in this FlowFile were invalid for the following reasons: The following 1 fields were missing: [[0]/Records/eventVersion]

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Nice. I like that, thank you Etienne. I also did a quick check and found this, which looks pretty recent, and pretty good... https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350 On Thu, Dec 5, 2019 at 9:04 AM Etienne Jouvin wrote: > On

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread Etienne Jouvin
On "official site" ;) https://jolt-demo.appspot.com/#inception Le jeu. 5 déc. 2019 à 15:01, James McMahon a écrit : > Absolutely. I am going to do that. When you started working with it, were > there any particularly helpful examples of its application you used to > learn it that you

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Absolutely. I am going to do that. When you started working with it, were there any particularly helpful examples of its application you used to learn it that you recommend? On Thu, Dec 5, 2019 at 8:57 AM Etienne Jouvin wrote: > Hello. > > You are right. If it works and you are satisfied, you

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread Etienne Jouvin
Hello. You are right. If it works and you are satisfied, you should keep your solution. By the wya JoltTransformation may be difficult at the very beginning. But it is very powerful and with some pratice, it begins to be easy. For study, you may give it a try. Regards. Etienne Jouvin Le jeu.

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Hello Etienne. Yes, Matt may have mentioned that approach and I started to look into it. My initial thought was this: is it much of a savings? My rudimentary process works in three process steps - each simple in configuration. The JoltTransformationJSON would eliminate only one processor, and it

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread Etienne Jouvin
Hello. Why don't you use a JoltTransformation process first to produce multiple element in JSON according value in the array, and duplicate common attributes for all. And then, you do the split. Etienne Le jeu. 5 déc. 2019 à 14:11, James McMahon a écrit : > Daeho and Matt, thank you for all

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Daeho and Matt, thank you for all your suggestions. You helped me get to a solution. Here is how I unwound my incoming JSON with a simple flow, My incoming JSON flowfile looks like this: { "KEY1":"value1", "KEY2":"value2", "FNAMES":["A","B","C","D"], "KEY4":2 } My goal is to have a

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread 노대호Daeho Ro
Hm I might wrong. It wouldn't preserve other keys, so you have to evaluate other keys first, and split FNAMES and evaluate again. Sorry for the confusion. 2019년 12월 5일 (목) 오후 8:29, James McMahon 님이 작성: > Typo in my initial reply. I did use $.FNAMES. It drops all the other > key/value pairs

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Typo in my initial reply. I did use $.FNAMES. It drops all the other key/value pairs in the output split result flowfiles. I configured my SplitJSON like so: JsonPathExpression$.FNAME*S* Null Value Representation empty string If there are two values in the json array for that key

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread 노대호Daeho Ro
Path to be $.FNAMES, that will work I guess. 2019년 12월 5일 (목) 오후 8:10, James McMahon 님이 작성: > I should add that I also tried this for JsonPathExpression $.* > That result also wasn't what I require, because it gave me 14 different > flowfiles each with only one value - - the two that resulted

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
I should add that I also tried this for JsonPathExpression $.* That result also wasn't what I require, because it gave me 14 different flowfiles each with only one value - - the two that resulted from the FNAME key, and one for each of the other 12 keys that had only one value. My incoming JSON

Re: Split a flow file to multiples, related by common key and with split count

2019-12-05 Thread James McMahon
Daeho, I configured my SplitJSON like so: JsonPathExpression$.FNAME Null Value Representation empty string If there are two values in the json array for that key FNAME, I do get two output flowfiles. But the only value present in the output is the value from the split of the list. All my