Re: How can I ExtractGrok from end-of-file?
Hello Eric, Have you found any solution for this? If your trailers (footer?) starts with certain byte sequence, then SplitContent may be helpful to split the content into Header+Payload, and the Trailers. If that works, then the subsequent flow can do something creative probably using RouteOnAttribute, GrokExtract, MergeContent (with defragment merge strategy) ... etc. Thanks, Koji On Fri, Mar 15, 2019 at 11:34 PM Eric Chaves wrote: > > Hi folks, > > I'm learning how to use grok with nifi starting with the ExtractGrok > processor and I'd like to use it to extract data from file headers and > trailers however since the GrokExtract processor only apply the grok > expression on the defined buffer size (and each of my file differs on size) I > can't evaluate trailers on every file. > > Any ideas on how could I apply the grok expression from the end of file > instead of from the beginning, or any alternative processor? > > Cheers, >
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Chad, I wanted to echo Bryan and say thanks for sharing the details of an upgrade process that works. Until we have improved NiFi to handle the upgrades regardless of order of steps, this is a helpful reference for the community for a sequence that can work. Cheers, Kevin On Tue, Mar 19, 2019 at 11:40 AM Bryan Bende wrote: > > Chad, > > Thanks for reporting your experience and glad to hear that there is a > good process to follow. > > We do want to make this situation better and there is a JIRA to track > the issue [1]. > > Thanks, > > Bryan > > [1] https://issues.apache.org/jira/browse/NIFI-6028 > > > > On Tue, Mar 19, 2019 at 11:23 AM Chad Woodhead wrote: > > > > Hey guys, > > > > I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was > > important to me. I've done some testing and wanted to share my results. > > > > Test environment setup: > > 2 NiFi Clusters > > 1 NiFi Registry > > > > Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 > > Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 > > > > Working upgrade method: > > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 > > 2. All Dev NiFi versioned flows showed local changes due to new properties > > in processors > > 3. Commit all Dev NiFi changes for all versioned process groups > > 4. In NiFi Registry, view and see all committed changes from Dev NiFi > > 5. On Cert NiFi, all versioned PGs show newer version available. > > 6. On Cert NiFi, change all versioned process groups to use new version. > > PGs now show "Flow version is current" (Note: the processors DO NOT become > > invalid and they do not show the new properties yet) > > 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1 > > 8. On Cert NiFi, all versioned process groups still show they are on the > > latest version and the processors DO show the new properties now > > > > Not working upgrade method: > > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 > > 2. All Dev NiFi versioned flows showed local changes due to new properties > > in processors > > 3. Commit all Dev NiFi changes to all versioned process groups > > 4. In NiFi Registry, view and see all committed changes from Dev NiFi > > 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1 > > 6. On Cert NiFi, versioned process groups show "Local changes have been > > made and a newer version of this flow is available". When clicking on PG > > and selecting "Version", only options are "Show local changes", "Revert > > local changes", and "Stop version control". > > "Show local changes" allows you to see what has changed (all new properties > > in processors). > > "Revert local changes" does nothing as these changes are required since > > they are new properties from the upgrade. > > 7. My only 2 options at this point are > > -Stop version control of the PG and start it back up, causing me to lose > > all history > > -Delete the PG on Cert and then re-import from NiFi Registry. This option > > really isn't an option when in Prod as I don't want to stop/delete > > production flows that need to keep ingesting data. > > > > So as shown above, when upgrading NiFi with versioned flows in NiFi > > Registry, the steps are very important and as long pull in the latest > > commit to Cert before upgrading Cert, your process groups will work as > > expected. > > > > Thanks, > > Chad > > > >> > >> > >> > >> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" wrote: > >> > >> *External Message* - Use caution before opening links or attachments > >> > >> Bryan, > >> > >> I agree, that is probably a solution. Unfortunately, there is no mass > >> upgrade option, so we'd have to manually touch ever versioned process > >> group (or scripted it). > >> > >> Thanks, > >> Peter > >> > >> -Original Message- > >> From: Bryan Bende [mailto:bbe...@gmail.com] > >> Sent: Thursday, November 29, 2018 1:29 PM > >> To: users@nifi.apache.org > >> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after > >> Processor Upgrades > >> > >> Peter, > >> > >> I feel like this came up before, and unfortunately I'm not sure there > >> is currently a solution. > >> > >> I think ultimately there needs to be some kind of force upgrade so you > >> can ignore the local changes and take whatever is available. > >> > >> The only thing I can think of, but haven't tried, is if you had > >> upgraded the PG in the second instance before upgrading NiFi itself, it > >> would bring in the new properties that are not valid in that version and > >> the processor would show as invalid, then upgrade NiFi and it would be > >> valid again. > >> > >> -Bryan > >> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) > >> wrote: > >> > > >> > Ran into a NiFi Registry issue while upgrading our instances to NiFi > >> 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so > >> after upgrading our, our versioned processor groups show as having local > >> changes, which
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Chad, Thanks for reporting your experience and glad to hear that there is a good process to follow. We do want to make this situation better and there is a JIRA to track the issue [1]. Thanks, Bryan [1] https://issues.apache.org/jira/browse/NIFI-6028 On Tue, Mar 19, 2019 at 11:23 AM Chad Woodhead wrote: > > Hey guys, > > I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was > important to me. I've done some testing and wanted to share my results. > > Test environment setup: > 2 NiFi Clusters > 1 NiFi Registry > > Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 > Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 > > Working upgrade method: > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 > 2. All Dev NiFi versioned flows showed local changes due to new properties in > processors > 3. Commit all Dev NiFi changes for all versioned process groups > 4. In NiFi Registry, view and see all committed changes from Dev NiFi > 5. On Cert NiFi, all versioned PGs show newer version available. > 6. On Cert NiFi, change all versioned process groups to use new version. PGs > now show "Flow version is current" (Note: the processors DO NOT become > invalid and they do not show the new properties yet) > 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1 > 8. On Cert NiFi, all versioned process groups still show they are on the > latest version and the processors DO show the new properties now > > Not working upgrade method: > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 > 2. All Dev NiFi versioned flows showed local changes due to new properties in > processors > 3. Commit all Dev NiFi changes to all versioned process groups > 4. In NiFi Registry, view and see all committed changes from Dev NiFi > 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1 > 6. On Cert NiFi, versioned process groups show "Local changes have been made > and a newer version of this flow is available". When clicking on PG and > selecting "Version", only options are "Show local changes", "Revert local > changes", and "Stop version control". > "Show local changes" allows you to see what has changed (all new properties > in processors). > "Revert local changes" does nothing as these changes are required since they > are new properties from the upgrade. > 7. My only 2 options at this point are > -Stop version control of the PG and start it back up, causing me to lose all > history > -Delete the PG on Cert and then re-import from NiFi Registry. This option > really isn't an option when in Prod as I don't want to stop/delete production > flows that need to keep ingesting data. > > So as shown above, when upgrading NiFi with versioned flows in NiFi Registry, > the steps are very important and as long pull in the latest commit to Cert > before upgrading Cert, your process groups will work as expected. > > Thanks, > Chad > >> >> >> >> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" wrote: >> >> *External Message* - Use caution before opening links or attachments >> >> Bryan, >> >> I agree, that is probably a solution. Unfortunately, there is no mass >> upgrade option, so we'd have to manually touch ever versioned process group >> (or scripted it). >> >> Thanks, >> Peter >> >> -Original Message- >> From: Bryan Bende [mailto:bbe...@gmail.com] >> Sent: Thursday, November 29, 2018 1:29 PM >> To: users@nifi.apache.org >> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after Processor >> Upgrades >> >> Peter, >> >> I feel like this came up before, and unfortunately I'm not sure there is >> currently a solution. >> >> I think ultimately there needs to be some kind of force upgrade so you >> can ignore the local changes and take whatever is available. >> >> The only thing I can think of, but haven't tried, is if you had upgraded >> the PG in the second instance before upgrading NiFi itself, it would bring >> in the new properties that are not valid in that version and the processor >> would show as invalid, then upgrade NiFi and it would be valid again. >> >> -Bryan >> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) >> wrote: >> > >> > Ran into a NiFi Registry issue while upgrading our instances to NiFi >> 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so >> after upgrading our, our versioned processor groups show as having local >> changes, which is good. We went ahead and checked the changes into the >> registry. >> > >> > >> > >> > Enter the second instance... we upgraded a second instance. It also >> see's local changes, but now the processor group is in conflict, because we >> have local (identical) changes, and we have a newer version checked in. If >> you try to revert the local changes so you can sync things up... you can't, >> because these are properties on the Processor, and the default values >> automatically come back. So our second processor group is in conflict and we
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Hey guys, I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was important to me. I've done some testing and wanted to share my results. *Test environment setup*: 2 NiFi Clusters 1 NiFi Registry Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0 *Working upgrade method*: 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 2. All Dev NiFi versioned flows showed local changes due to new properties in processors 3. Commit all Dev NiFi changes for all versioned process groups 4. In NiFi Registry, view and see all committed changes from Dev NiFi 5. On Cert NiFi, all versioned PGs show newer version available. 6. On Cert NiFi, change all versioned process groups to use new version. PGs now show "Flow version is current" (Note: the processors DO NOT become invalid and they do not show the new properties yet) 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1 8. On Cert NiFi, all versioned process groups still show they are on the latest version and the processors DO show the new properties now *Not working upgrade method*: 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1 2. All Dev NiFi versioned flows showed local changes due to new properties in processors 3. Commit all Dev NiFi changes to all versioned process groups 4. In NiFi Registry, view and see all committed changes from Dev NiFi 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1 6. On Cert NiFi, versioned process groups show "Local changes have been made and a newer version of this flow is available". When clicking on PG and selecting "Version", only options are "Show local changes", "Revert local changes", and "Stop version control". "Show local changes" allows you to see what has changed (all new properties in processors). "Revert local changes" does nothing as these changes are required since they are new properties from the upgrade. 7. My only 2 options at this point are -Stop version control of the PG and start it back up, causing me to lose all history -Delete the PG on Cert and then re-import from NiFi Registry. This option really isn't an option when in Prod as I don't want to stop/delete production flows that need to keep ingesting data. So as shown above, when upgrading NiFi with versioned flows in NiFi Registry, the steps are very important and as long pull in the latest commit to Cert before upgrading Cert, your process groups will work as expected. Thanks, Chad > > > On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" wrote: > > *External Message* - Use caution before opening links or attachments > > Bryan, > > I agree, that is probably a solution. Unfortunately, there is no mass > upgrade option, so we'd have to manually touch ever versioned process group > (or scripted it). > > Thanks, > Peter > > -Original Message- > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Thursday, November 29, 2018 1:29 PM > To: users@nifi.apache.org > Subject: [EXT] Re: Problems with NiFi Registry Conflicts after > Processor Upgrades > > Peter, > > I feel like this came up before, and unfortunately I'm not sure there > is currently a solution. > > I think ultimately there needs to be some kind of force upgrade so you > can ignore the local changes and take whatever is available. > > The only thing I can think of, but haven't tried, is if you had > upgraded the PG in the second instance before upgrading NiFi itself, it > would bring in the new properties that are not valid in that version and > the processor would show as invalid, then upgrade NiFi and it would be > valid again. > > -Bryan > On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) < > pwi...@micron.com> wrote: > > > > Ran into a NiFi Registry issue while upgrading our instances to NiFi > 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so > after upgrading our, our versioned processor groups show as having local > changes, which is good. We went ahead and checked the changes into the > registry. > > > > > > > > Enter the second instance... we upgraded a second instance. It also > see's local changes, but now the processor group is in conflict, because we > have local (identical) changes, and we have a newer version checked in. If > you try to revert the local changes so you can sync things up... you can't, > because these are properties on the Processor, and the default values > automatically come back. So our second processor group is in conflict and > we haven't found a way to bring it back in sync without deleting it and re > loading it from the registry. Help would be appreciated. > > > > > > > > Thanks, > > > > Peter > > >
Re: Can't trim leading whitespaces with UpdateRecord
Hi Eric,
I did some quick tests, replaceRegex works well for me. The regex patterns
should behave as expected (see:
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/functions/ReplaceRegex.java#L49-L49),
I don't see any reason why it doesn't work for you.
Side note: if you want to trim both leading and trailing white spaces I'd
suggest to use the trim() function: switch to Literal Value replacement
strategy and use "${field.value:trim()}", my gut feeling is that it'd
perform better.
Best,
Denes
On Sun, Mar 17, 2019 at 8:10 PM Eric Chaves wrote:
> Hi Folks,
>
> Forgive me for this dummy question but I've been cracking my head for
> hours.
>
> Is there anything special in RecordPath regular expressions? I'm trying
> to trim whitespaces by using UpdateRecord (with replacement strategy of
> Record Path Value ) replaceRegex over current field value but I can't find
> any expression that works.
>
> All my attempts produce an output with the original value however simple
> replacements like replaceRegex(/FULL_NAME,'JON', 'ERIC' ) works. I've
> already tried expressions like:
>
> replaceRegex(/FULL_NAME,'\s', '' )
> replaceRegex(/FULL_NAME,'\s+', '-' )
> replaceRegex(/FULL_NAME,'\s{2,}', '-' )
> replaceRegex(/SSN,'(\d+)', '$1' )
> replaceRegex(/SSN,'\s+$', '' )
>
> If I try the regex in a java regex tester (like
> https://www.freeformatter.com/java-regex-tester.html) all expressions
> matches/replaces as expected but in the UpdateRecord processor they have no
> effect.
>
> What am I missing?
>
> Thanks
>
