I've achieved this sort of thing previously with UpdateRecord by using the matchesRegex[1] function as a predicate for the fields to be selected, i.e. include a regex match predicate[2] as part of the dynamic property key, then use a matching format in the property's value.
It gets a little cumbersome when you've multiple formats to handle though. You can do a similar thing with chained matches and ifElse functions in Expression Language (if working with attributes). An alternative for 1.11.4 might be to use something like ExecuteGroovyScript to script something for processing the different formats, i.e. code the if-else matches into a script. You need to be careful not to read too much data into memory at once though when using such scripts with large files. Unfortunately, I've moved away from having to use the above now (we're created a custom processor that does all of the above in Java along with a bunch of other stuff), so don't have a ready example to hand. [1] https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html#matchesregex [2] https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html#predicates --- *Chris Sampson* IT Consultant [email protected] <https://www.naimuri.com/> On Thu, 25 Feb 2021 at 17:11, Mark Payne <[email protected]> wrote: > Greg, > > Yeah, that processor came out in 1.12. Jolt transform might work, but I am > not very familiar with Jolt so I can’t really say for sure. > > Thanks > -Mark > > > On Feb 25, 2021, at 11:48 AM, Gregory M. Foreman < > [email protected]> wrote: > > > > Mark: > > > > Thanks, but it appears that the processor is not available in Nifi > 1.11.4. > > > > Would a custom Jolt transform make sense? > > > > Greg > > > > > >> On Feb 25, 2021, at 10:01 AM, Mark Payne <[email protected]> wrote: > >> > >> Greg, > >> > >> I would probably ScriptedTransformRecord in this case and script a > solution that looks at the value and determines which pattern to use. > >> > >> Thanks > >> -Mark > >> > >>> On Feb 25, 2021, at 9:35 AM, Gregory M. Foreman < > [email protected]> wrote: > >>> > >>> Hello, > >>> > >>> I am attempting to parse timestamps of various formats/precision > within the same field in json. Here is an example (using Nifi 1.11.4): > >>> > >>> input: > >>> > >>> {"cd": "2021-02-24T12:01:01"} > >>> {"cd": "2021-02-24T12:01:01Z"} > >>> {"cd": "2021-02-25T12:01:01.99"} > >>> {"cd": "2021-02-26T12:01:01.999"} > >>> > >>> conversion in UpdateRecord: > >>> > >>> /cd = format(toDate(/cd, "yyyy-MM-dd'T'HH:mm:ss"), > "yyyy-MM-dd'T'HH:mm:ss.SSSZZ") > >>> > >>> output: > >>> > >>> {"cd":"2021-02-24T12:01:01.000+0000"} > >>> {"cd":"2021-02-24T12:01:01.000+0000"} > >>> {"cd":"2021-02-25T12:01:01.000+0000"} << precision lost > >>> {"cd":"2021-02-26T12:01:01.000+0000"} << precision lost > >>> > >>> changing the conversion in UpdateRecord to this: > >>> > >>> /cd = format(toDate(/cd, "yyyy-MM-dd'T'HH:mm:ss.SSS"), > "yyyy-MM-dd'T'HH:mm:ss.SSSZZ") > >>> > >>> output: > >>> > >>> {"cd":"2021-02-24T12:01:01"} << skipped > >>> {"cd":"2021-02-24T12:01:01Z"} << skipped > >>> {"cd":"2021-02-25T12:01:01.099+0000"} << not sure if this is correct > >>> {"cd":"2021-02-26T12:01:01.999+0000"} << correct > >>> > >>> > >>> Is there a way to change the timestamp parsing format on a per-record > basis? > >>> > >>> Thanks, > >>> Greg > >>> > >> > > > >
