Re: Setting zookeeper.sasl.client=false

[Chad Woodhead]

Bryan,

I don’t have anything set for nifi.zookeeper.auth.type, so then by default by 
the Z-Nodes aren’t created with SASL and therefore have ACL open to everyone. I 
agree, in my cluster setting zookeeper.sasl.client=false won’t affect my 
current setup and flows. Wanted to make sure I wasn’t breaking anything and 
also understood correctly that if I needed to one day connect to Z-Nodes that 
do have SASL, I would then impact the flows that connect to Z-Nodes without 
SASL.

I have a feeling you are correct (haven’t had time to test either) that if 
nifi.zookeeper.auth.type=sasl and zookeeper.sasl.client=false, NiFi would fail 
to read its own Z-Nodes.

-Chad

> On Aug 2, 2019, at 2:57 PM, Bryan Bende  wrote:
> 
> Chad,
> 
> I was looking into something related to this recently and I think your
> description is accurate. Unfortunately ZooKeeper client relies heavily
> on system properties which isn't great for talking to a bunch of
> different systems like NiFi.
> 
> One thing I would be curious about, what value do you have in
> nifi.properties for nifi.zookeeper.auth.type= ?
> 
> If it is not set, or is set to default, then the Z-Nodes created by
> NiFi would not be created with SASL and would have an ACL open to
> everyone, so then setting zookeeper.sasl.client=false probably doesn't
> impact anything with your NiFi cluster.
> 
> If nifi.zookeeper.auth.type=sasl then I wonder if you set
> zookeeper.sasl.client=false, would your NiFi cluster fail to read its
> own Z-Nodes on next restart?
> 
> I've been wanting to try this for a while, but haven't had time.
> 
> -Bryan
> 
> On Fri, Aug 2, 2019 at 3:09 PM Chad Woodhead  wrote:
>> 
>> I’m building a flow that uses ExecuteSQL to query data from Phoenix on top 
>> of a Kerberized Ambari Metrics’ HBase (it hits the AMS Zookeeper). I ran 
>> into issues and the logs showed NiFi (also kerberized) was getting auth 
>> failed when connecting to ZNode.
>> 
>> To confirm I had all my proper AMS conf files and phoenix jars, I tried 
>> tweaking the DBCP and customizing the phoenix-client.jar for this specific 
>> Hbase (using many links online of people doing this same thing), but 
>> unfortunately didn’t resolve the Znode error.
>> 
>> I realized that by default AMS does not create the Znode on Zookeeper secure 
>> with SASL, and by default NiFi has zookeeper.sasl.client=true causing NiFi 
>> to use SASL for zookeeper client connections. So I tested setting 
>> ‘java.arg.X=-Dzookeeper.sasl.client=false’ in my bootstrap.conf file and 
>> finally NiFi was able to successfully connect and query the data.
>> 
>> I don’t have much experience with SASL and Znodes, so I wanted to know if 
>> there are any issues I can run into by setting it to false? Or even just 
>> going against security recommendations?
>> 
>> Am I understanding it correctly that if zookeeper.sasl.client=true then NiFi 
>> can ONLY connect to Znodes that use SASL, and if zookeeper.sasl.client=false 
>> then NiFi can ONLY connect to Znodes that do not use SASL?
>> 
>> Any help would be appreciated!
>> 
>> Thanks,
>> Chad

Setting zookeeper.sasl.client=false

[Chad Woodhead]

I’m building a flow that uses ExecuteSQL to query data from Phoenix on top of a 
Kerberized Ambari Metrics’ HBase (it hits the AMS Zookeeper). I ran into issues 
and the logs showed NiFi (also kerberized) was getting auth failed when 
connecting to ZNode. 

To confirm I had all my proper AMS conf files and phoenix jars, I tried 
tweaking the DBCP and customizing the phoenix-client.jar for this specific 
Hbase (using many links online of people doing this same thing), but 
unfortunately didn’t resolve the Znode error. 

I realized that by default AMS does not create the Znode on Zookeeper secure 
with SASL, and by default NiFi has zookeeper.sasl.client=true causing NiFi to 
use SASL for zookeeper client connections. So I tested setting 
‘java.arg.X=-Dzookeeper.sasl.client=false’ in my bootstrap.conf file and 
finally NiFi was able to successfully connect and query the data.

I don’t have much experience with SASL and Znodes, so I wanted to know if there 
are any issues I can run into by setting it to false? Or even just going 
against security recommendations?

Am I understanding it correctly that if zookeeper.sasl.client=true then NiFi 
can ONLY connect to Znodes that use SASL, and if zookeeper.sasl.client=false 
then NiFi can ONLY connect to Znodes that do not use SASL?

Any help would be appreciated!

Thanks,
Chad

Re: Problems with NiFi Registry Conflicts after Processor Upgrades

[Chad Woodhead]

Hey guys,

I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was
important to me. I've done some testing and wanted to share my results.

*Test environment setup*:
2 NiFi Clusters
1 NiFi Registry

Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0

*Working upgrade method*:
1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
2. All Dev NiFi versioned flows showed local changes due to new properties
in processors
3. Commit all Dev NiFi changes for all versioned process groups
4. In NiFi Registry, view and see all committed changes from Dev NiFi
5. On Cert NiFi, all versioned PGs show newer version available.
6. On Cert NiFi, change all versioned process groups to use new version.
PGs now show "Flow version is current" (Note: the processors DO NOT become
invalid and they do not show the new properties yet)
7. Upgrade Cert NiFi from 1.8.0 to 1.9.1
8. On Cert NiFi, all versioned process groups still show they are on the
latest version and the processors DO show the new properties now

*Not working upgrade method*:
1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
2. All Dev NiFi versioned flows showed local changes due to new properties
in processors
3. Commit all Dev NiFi changes to all versioned process groups
4. In NiFi Registry, view and see all committed changes from Dev NiFi
5. Upgrade Cert NiFi from 1.8.0 to 1.9.1
6. On Cert NiFi, versioned process groups show "Local changes have been
made and a newer version of this flow is available". When clicking on PG
and selecting "Version", only options are "Show local changes", "Revert
local changes", and "Stop version control".
"Show local changes" allows you to see what has changed (all new properties
in processors).
"Revert local changes" does nothing as these changes are required since
they are new properties from the upgrade.
7. My only 2 options at this point are
-Stop version control of the PG and start it back up, causing me to lose
all history
-Delete the PG on Cert and then re-import from NiFi Registry. This option
really isn't an option when in Prod as I don't want to stop/delete
production flows that need to keep ingesting data.

So as shown above, when upgrading NiFi with versioned flows in NiFi
Registry, the steps are very important and as long pull in the latest
commit to Cert before upgrading Cert, your process groups will work as
expected.

Thanks,
Chad


>
>
> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)"  wrote:
>
> *External Message* - Use caution before opening links or attachments
>
> Bryan,
>
> I agree, that is probably a solution. Unfortunately, there is no mass
> upgrade option, so we'd have to manually touch ever versioned process group
> (or scripted it).
>
> Thanks,
>   Peter
>
> -Original Message-
> From: Bryan Bende [mailto:bbe...@gmail.com]
> Sent: Thursday, November 29, 2018 1:29 PM
> To: users@nifi.apache.org
> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after
> Processor Upgrades
>
> Peter,
>
> I feel like this came up before, and unfortunately I'm not sure there
> is currently a solution.
>
> I think ultimately there needs to be some kind of force upgrade so you
> can ignore the local changes and take whatever is available.
>
> The only thing I can think of, but haven't tried, is if you had
> upgraded the PG in the second instance before upgrading NiFi itself, it
> would bring in the new properties that are not valid in that version and
> the processor would show as invalid, then upgrade NiFi and it would be
> valid again.
>
> -Bryan
> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) <
> pwi...@micron.com> wrote:
> >
> > Ran into a NiFi Registry issue while upgrading our instances to NiFi
> 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so
> after upgrading our, our versioned processor groups show as having local
> changes, which is good. We went ahead and checked the changes into the
> registry.
> >
> >
> >
> > Enter the second instance... we upgraded a second instance. It also
> see's local changes, but now the processor group is in conflict, because we
> have local (identical) changes, and we have a newer version checked in. If
> you try to revert the local changes so you can sync things up... you can't,
> because these are properties on the Processor, and the default values
> automatically come back. So our second processor group is in conflict and
> we haven't found a way to bring it back in sync without deleting it and re
> loading it from the registry. Help would be appreciated.
> >
> >
> >
> > Thanks,
> >
> >   Peter
>
>
>

Re: Different NiFi Node sizes within same cluster

[Chad Woodhead]

Thanks all for the input. So from what I'm gathering, storage differences
of around 5 GB (125 GB vs 130 GB) should not cause any problems/load
impacts. Larger storage differences could have load impacts. Differences in
CPU and RAM could definitely have load impacts. Luckily my older nodes have
the same CPU and RAM counts/specs as my new nodes.

The last thing I'm looking to understand is what Byran B brought up, do
load balanced connections take into consideration the load of each node?

Thanks,
Chad

On Wed, Mar 6, 2019 at 4:50 PM Bryan Bende  wrote:

> Yea ListenTCP also doesn't handle the back-pressure with the client
> the way it really should.
>
> Regarding the load balancing, I believe traditional s2s does factor in
> the load of each node when deciding how to load balance, but I don't
> know if this is part of load balanced connections or not. Mark P would
> know for sure.
>
> On Wed, Mar 6, 2019 at 4:47 PM James Srinivasan
>  wrote:
> >
> > Yup, but because of the unfortunate way the source (outside NiFi)
> > works, it doesn't buffer for long when the connection doesn't pull or
> > drops. It behaves far more like a 5 Mbps UDP stream really :-(
> >
> > On Wed, 6 Mar 2019 at 21:44, Bryan Bende  wrote:
> > >
> > > James, just curious, what was your source processor in this case?
> ListenTCP?
> > >
> > > On Wed, Mar 6, 2019 at 4:26 PM Jon Logan  wrote:
> > > >
> > > > What really would resolve some of these issues is backpressure on
> CPU -- ie. let Nifi throttle itself down to not choke the machine until it
> dies if constrained on CPU. Easier said than done unfortunately.
> > > >
> > > > On Wed, Mar 6, 2019 at 4:23 PM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
> > > >>
> > > >> In our case, backpressure applied all the way up to the TCP network
> > > >> source which meant we lost data. AIUI, current load balancing is
> round
> > > >> robin (and two other options prob not relevant). Would actual load
> > > >> balancing (e.g. send to node with lowest OS load, or number of
> active
> > > >> threads) be a reasonable request?
> > > >>
> > > >> On Wed, 6 Mar 2019 at 20:51, Joe Witt  wrote:
> > > >> >
> > > >> > This is generally workable (heterogenous node capabilities) in
> NiFi clustering.  But you do want to leverage back-pressure and load
> balanced connections so that faster nodes will have an opportunity to take
> on the workload for slower nodes.
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> > On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
> > > >> >>
> > > >> >> Yes, we hit this with the new load balanced queues (which, to be
> fair, we also had with remote process groups previously). Two "old" nodes
> got saturated and their queues filled while three "new" nodes were fine.
> > > >> >>
> > > >> >> My "solution" was to move everything to new hardware which we
> had inbound anyway.
> > > >> >>
> > > >> >> On Wed, 6 Mar 2019, 20:40 Jon Logan, 
> wrote:
> > > >> >>>
> > > >> >>> You may run into issues with different processing power, as
> some machines may be overwhelmed in order to saturate other machines.
> > > >> >>>
> > > >> >>> On Wed, Mar 6, 2019 at 3:34 PM Mark Payne 
> wrote:
> > > >> >>>>
> > > >> >>>> Chad,
> > > >> >>>>
> > > >> >>>> This should not be a problem, given that all nodes have enough
> storage available to handle the influx of data.
> > > >> >>>>
> > > >> >>>> Thanks
> > > >> >>>> -Mark
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> > On Mar 6, 2019, at 1:44 PM, Chad Woodhead <
> chadwoodh...@gmail.com> wrote:
> > > >> >>>> >
> > > >> >>>> > Are there any negative effects of having filesystem mounts
> (dedicated mounts for each repo) used by the different NiFi repositories
> differ in size on NiFi nodes within the same cluster? For instance, if some
> nodes have a content_repo mount of 130 GB and other nodes have a
> content_repo mount of 125 GB, could that cause any problems or cause one
> node to be used more since it has more space? What about if the difference
> was larger, by say a 100 GB difference?
> > > >> >>>> >
> > > >> >>>> > Trying to repurpose old nodes and add them as NiFi nodes,
> but their mount sizes are different than my current cluster’s nodes and
> I’ve noticed I can’t set the max size limit to use of a particular mount
> for a repo.
> > > >> >>>> >
> > > >> >>>> > -Chad
> > > >> >>>>
>

Different NiFi Node sizes within same cluster

[Chad Woodhead]

Are there any negative effects of having filesystem mounts (dedicated
mounts for each repo) used by the different NiFi repositories differ in
size on NiFi nodes within the same cluster? For instance, if some nodes
have a content_repo mount of 130 GB and other nodes have a content_repo
mount of 125 GB, could that cause any problems or cause one node to be used
more since it has more space? What about if the difference was larger, by
say a 100 GB difference?

Trying to repurpose old nodes and add them as NiFi nodes, but their mount
sizes are different than my current cluster’s nodes and I’ve noticed I
can’t set the max size limit to use of a particular mount for a repo.

-Chad

Re: Versioned flows not maintaining Load Balance Strategy

[Chad Woodhead]

Bryan,

Wanted to update you. Spoke with Hortonworks and they informed me the
version definitions included in the HDF 3.3.x versions used NiFi Registry
0.3.0 because 0.3.0 was supposed to go in those releases. Unfortunately it
was discovered that NiFi Registry was pointed at the wrong repo and pulled
0.2.0 but kept the label 0.3.0. So I'm actually running 0.2.0 and this
explains the behavior I'm seeing.

Thanks for the help and hopefully this helps anyone use running HDF and
seeing this behavior.

-Chad

On Tue, Feb 19, 2019 at 4:02 PM Chad Woodhead 
wrote:

> Bryan,
>
> Thanks for the update. I will reach out to Hortonworks to see what could
> be happening. I'll update this thread with what I find.
>
> -Chad
>
> On Tue, Feb 19, 2019 at 3:49 PM Bryan Bende  wrote:
>
>> Chad,
>>
>> I suspect it may be an issue with the version of NiFi registry in the
>> vendor distribution.
>>
>> As far as I can tell, it is working correctly on apache releases of
>> NiFi 1.8.0/1.9.0-RC2 and registry 0.3.0.
>>
>> -Bryan
>>
>> On Tue, Feb 19, 2019 at 3:27 PM Chad Woodhead 
>> wrote:
>> >
>> > Bryan,
>> >
>> > I also just tested with a new flow not in version control with NiFi
>> Registry and then started version control with it, and same behavior on my
>> side. No load balance connection properties in the snapshot.
>> >
>> > Thanks,
>> > Chad
>> >
>> > On Tue, Feb 19, 2019 at 3:21 PM Chad Woodhead 
>> wrote:
>> >>
>> >> Bryan,
>> >>
>> >> No the snapshot in my git repo (we use GitFlowPersistenceProvider)
>> does not have the load balance connection properties. If I create a
>> template of the same PG on NiFi, I do see the load balance connection
>> properties in the xml. So for some reason the load balance connection
>> properties are not being sent to NiFi Registry.
>> >>
>> >> I am running HDF 3.3.0.0-165
>> >>
>> >> Thanks,
>> >> Chad
>> >>
>> >>
>> >> On Tue, Feb 19, 2019 at 1:08 PM Bryan Bende  wrote:
>> >>>
>> >>> Chad,
>> >>>
>> >>> Using the 1.9.0-RC2 build of NiFi and the 0.3.0 release of registry, I
>> >>> haven't been able to reproduce the issue.
>> >>>
>> >>> I don't know of anything that would have fixed the issue between 1.8.0
>> >>> and 1.9.0, so I'm not sure what you are running into.
>> >>>
>> >>> Can you look in your registry flow_storage directory (or git repo) and
>> >>> find the flow snapshot file for the latest snapshot that you believe
>> >>> has the load balanced connection, and then look for something like:
>> >>>
>> >>> "connections" : [ {
>> >>>   "backPressureDataSizeThreshold" : "1 GB",
>> >>>   "backPressureObjectThreshold" : 1,
>> >>>   "bends" : [ ],
>> >>>   "componentType" : "CONNECTION",
>> >>>   "destination" : {
>> >>> "comments" : "",
>> >>> "groupId" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
>> >>> "id" : "8a6087bd-d422-331e-9bc7-d7d2f533cfc0",
>> >>> "name" : "LogAttribute",
>> >>> "type" : "PROCESSOR"
>> >>>   },
>> >>>   "flowFileExpiration" : "0 sec",
>> >>>   "groupIdentifier" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
>> >>>   "identifier" : "ea1b818e-1d2e-3c42-a956-1796392e85be",
>> >>>   "labelIndex" : 1,
>> >>>   "loadBalanceCompression" : "DO_NOT_COMPRESS",
>> >>>   "loadBalanceStrategy" : "ROUND_ROBIN",
>> >>>
>> >>> If using the file based storage then the paths in flow_storage are
>> >>> ///.snapshot.
>> >>>
>> >>> If you don't see those last two load balanced related fields then that
>> >>> would cause the issue, but not sure why they wouldn't be populated
>> >>> from the NiFi side.
>> >>>
>> >>> Also, can you clarify if you are using apache releases, or a vendor
>> >>> release such as HDF?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Bryan
>> >>>
>> >>> On Mon, Feb 18, 2019 at 7:49 AM Chad Woodhead 
>> wrote:
>> >>> >
>> >>> > I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed
>> load balance strategies on queues aren't coming through versioned flows in
>> NiFi Registry. Here are the steps I am performing:
>> >>> >
>> >>> > 1. Have existing flow running on latest flow version in Dev and
>> Cert (flow has already been developed and in version control)
>> >>> > 2. Add load balance strategy to queue in Dev flow
>> >>> > 3. NiFi shows local changes and I commit the changes to NiFi
>> Registry
>> >>> > 4. Cert NIFi shows new flow version
>> >>> > 5. Pull latest version down to Cert. The load balance strategy for
>> the queue doesn't come with it. I then have to edit flow on Cert to add the
>> load balance strategy for the queue which causes NiFi to see local changes
>> which I then have to commit to Registry again but this time from Cert.
>> >>> >
>> >>> > I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194
>> which made me think I shouldn't be experiencing the behavior I am seeing.
>> >>> >
>> >>> > Thanks,
>> >>> > Chad
>>
>

Re: Versioned flows not maintaining Load Balance Strategy

[Chad Woodhead]

Bryan,

Thanks for the update. I will reach out to Hortonworks to see what could be
happening. I'll update this thread with what I find.

-Chad

On Tue, Feb 19, 2019 at 3:49 PM Bryan Bende  wrote:

> Chad,
>
> I suspect it may be an issue with the version of NiFi registry in the
> vendor distribution.
>
> As far as I can tell, it is working correctly on apache releases of
> NiFi 1.8.0/1.9.0-RC2 and registry 0.3.0.
>
> -Bryan
>
> On Tue, Feb 19, 2019 at 3:27 PM Chad Woodhead 
> wrote:
> >
> > Bryan,
> >
> > I also just tested with a new flow not in version control with NiFi
> Registry and then started version control with it, and same behavior on my
> side. No load balance connection properties in the snapshot.
> >
> > Thanks,
> > Chad
> >
> > On Tue, Feb 19, 2019 at 3:21 PM Chad Woodhead 
> wrote:
> >>
> >> Bryan,
> >>
> >> No the snapshot in my git repo (we use GitFlowPersistenceProvider) does
> not have the load balance connection properties. If I create a template of
> the same PG on NiFi, I do see the load balance connection properties in the
> xml. So for some reason the load balance connection properties are not
> being sent to NiFi Registry.
> >>
> >> I am running HDF 3.3.0.0-165
> >>
> >> Thanks,
> >> Chad
> >>
> >>
> >> On Tue, Feb 19, 2019 at 1:08 PM Bryan Bende  wrote:
> >>>
> >>> Chad,
> >>>
> >>> Using the 1.9.0-RC2 build of NiFi and the 0.3.0 release of registry, I
> >>> haven't been able to reproduce the issue.
> >>>
> >>> I don't know of anything that would have fixed the issue between 1.8.0
> >>> and 1.9.0, so I'm not sure what you are running into.
> >>>
> >>> Can you look in your registry flow_storage directory (or git repo) and
> >>> find the flow snapshot file for the latest snapshot that you believe
> >>> has the load balanced connection, and then look for something like:
> >>>
> >>> "connections" : [ {
> >>>   "backPressureDataSizeThreshold" : "1 GB",
> >>>   "backPressureObjectThreshold" : 1,
> >>>   "bends" : [ ],
> >>>   "componentType" : "CONNECTION",
> >>>   "destination" : {
> >>> "comments" : "",
> >>> "groupId" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
> >>> "id" : "8a6087bd-d422-331e-9bc7-d7d2f533cfc0",
> >>> "name" : "LogAttribute",
> >>> "type" : "PROCESSOR"
> >>>   },
> >>>   "flowFileExpiration" : "0 sec",
> >>>   "groupIdentifier" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
> >>>   "identifier" : "ea1b818e-1d2e-3c42-a956-1796392e85be",
> >>>   "labelIndex" : 1,
> >>>   "loadBalanceCompression" : "DO_NOT_COMPRESS",
> >>>   "loadBalanceStrategy" : "ROUND_ROBIN",
> >>>
> >>> If using the file based storage then the paths in flow_storage are
> >>> ///.snapshot.
> >>>
> >>> If you don't see those last two load balanced related fields then that
> >>> would cause the issue, but not sure why they wouldn't be populated
> >>> from the NiFi side.
> >>>
> >>> Also, can you clarify if you are using apache releases, or a vendor
> >>> release such as HDF?
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>> On Mon, Feb 18, 2019 at 7:49 AM Chad Woodhead 
> wrote:
> >>> >
> >>> > I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed load
> balance strategies on queues aren't coming through versioned flows in NiFi
> Registry. Here are the steps I am performing:
> >>> >
> >>> > 1. Have existing flow running on latest flow version in Dev and Cert
> (flow has already been developed and in version control)
> >>> > 2. Add load balance strategy to queue in Dev flow
> >>> > 3. NiFi shows local changes and I commit the changes to NiFi Registry
> >>> > 4. Cert NIFi shows new flow version
> >>> > 5. Pull latest version down to Cert. The load balance strategy for
> the queue doesn't come with it. I then have to edit flow on Cert to add the
> load balance strategy for the queue which causes NiFi to see local changes
> which I then have to commit to Registry again but this time from Cert.
> >>> >
> >>> > I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194
> which made me think I shouldn't be experiencing the behavior I am seeing.
> >>> >
> >>> > Thanks,
> >>> > Chad
>

Re: Versioned flows not maintaining Load Balance Strategy

[Chad Woodhead]

Bryan,

I also just tested with a new flow not in version control with NiFi
Registry and then started version control with it, and same behavior on my
side. No load balance connection properties in the snapshot.

Thanks,
Chad

On Tue, Feb 19, 2019 at 3:21 PM Chad Woodhead 
wrote:

> Bryan,
>
> No the snapshot in my git repo (we use GitFlowPersistenceProvider) does
> not have the load balance connection properties. If I create a template of
> the same PG on NiFi, I do see the load balance connection properties in the
> xml. So for some reason the load balance connection properties are not
> being sent to NiFi Registry.
>
> I am running HDF 3.3.0.0-165
>
> Thanks,
> Chad
>
>
> On Tue, Feb 19, 2019 at 1:08 PM Bryan Bende  wrote:
>
>> Chad,
>>
>> Using the 1.9.0-RC2 build of NiFi and the 0.3.0 release of registry, I
>> haven't been able to reproduce the issue.
>>
>> I don't know of anything that would have fixed the issue between 1.8.0
>> and 1.9.0, so I'm not sure what you are running into.
>>
>> Can you look in your registry flow_storage directory (or git repo) and
>> find the flow snapshot file for the latest snapshot that you believe
>> has the load balanced connection, and then look for something like:
>>
>> "connections" : [ {
>>   "backPressureDataSizeThreshold" : "1 GB",
>>   "backPressureObjectThreshold" : 1,
>>   "bends" : [ ],
>>   "componentType" : "CONNECTION",
>>   "destination" : {
>> "comments" : "",
>> "groupId" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
>> "id" : "8a6087bd-d422-331e-9bc7-d7d2f533cfc0",
>> "name" : "LogAttribute",
>> "type" : "PROCESSOR"
>>   },
>>   "flowFileExpiration" : "0 sec",
>>   "groupIdentifier" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
>>   "identifier" : "ea1b818e-1d2e-3c42-a956-1796392e85be",
>>   "labelIndex" : 1,
>>   "loadBalanceCompression" : "DO_NOT_COMPRESS",
>>   "loadBalanceStrategy" : "ROUND_ROBIN",
>>
>> If using the file based storage then the paths in flow_storage are
>> ///.snapshot.
>>
>> If you don't see those last two load balanced related fields then that
>> would cause the issue, but not sure why they wouldn't be populated
>> from the NiFi side.
>>
>> Also, can you clarify if you are using apache releases, or a vendor
>> release such as HDF?
>>
>> Thanks,
>>
>> Bryan
>>
>> On Mon, Feb 18, 2019 at 7:49 AM Chad Woodhead 
>> wrote:
>> >
>> > I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed load
>> balance strategies on queues aren't coming through versioned flows in NiFi
>> Registry. Here are the steps I am performing:
>> >
>> > 1. Have existing flow running on latest flow version in Dev and Cert
>> (flow has already been developed and in version control)
>> > 2. Add load balance strategy to queue in Dev flow
>> > 3. NiFi shows local changes and I commit the changes to NiFi Registry
>> > 4. Cert NIFi shows new flow version
>> > 5. Pull latest version down to Cert. The load balance strategy for the
>> queue doesn't come with it. I then have to edit flow on Cert to add the
>> load balance strategy for the queue which causes NiFi to see local changes
>> which I then have to commit to Registry again but this time from Cert.
>> >
>> > I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194
>> which made me think I shouldn't be experiencing the behavior I am seeing.
>> >
>> > Thanks,
>> > Chad
>>
>

Re: Versioned flows not maintaining Load Balance Strategy

[Chad Woodhead]

Bryan,

No the snapshot in my git repo (we use GitFlowPersistenceProvider) does not
have the load balance connection properties. If I create a template of the
same PG on NiFi, I do see the load balance connection properties in the
xml. So for some reason the load balance connection properties are not
being sent to NiFi Registry.

I am running HDF 3.3.0.0-165

Thanks,
Chad


On Tue, Feb 19, 2019 at 1:08 PM Bryan Bende  wrote:

> Chad,
>
> Using the 1.9.0-RC2 build of NiFi and the 0.3.0 release of registry, I
> haven't been able to reproduce the issue.
>
> I don't know of anything that would have fixed the issue between 1.8.0
> and 1.9.0, so I'm not sure what you are running into.
>
> Can you look in your registry flow_storage directory (or git repo) and
> find the flow snapshot file for the latest snapshot that you believe
> has the load balanced connection, and then look for something like:
>
> "connections" : [ {
>   "backPressureDataSizeThreshold" : "1 GB",
>   "backPressureObjectThreshold" : 1,
>   "bends" : [ ],
>   "componentType" : "CONNECTION",
>   "destination" : {
> "comments" : "",
> "groupId" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
> "id" : "8a6087bd-d422-331e-9bc7-d7d2f533cfc0",
> "name" : "LogAttribute",
> "type" : "PROCESSOR"
>   },
>   "flowFileExpiration" : "0 sec",
>   "groupIdentifier" : "3320ad51-dbb1-388e-8457-38e1fe226e2e",
>   "identifier" : "ea1b818e-1d2e-3c42-a956-1796392e85be",
>   "labelIndex" : 1,
>   "loadBalanceCompression" : "DO_NOT_COMPRESS",
>   "loadBalanceStrategy" : "ROUND_ROBIN",
>
> If using the file based storage then the paths in flow_storage are
> ///.snapshot.
>
> If you don't see those last two load balanced related fields then that
> would cause the issue, but not sure why they wouldn't be populated
> from the NiFi side.
>
> Also, can you clarify if you are using apache releases, or a vendor
> release such as HDF?
>
> Thanks,
>
> Bryan
>
> On Mon, Feb 18, 2019 at 7:49 AM Chad Woodhead 
> wrote:
> >
> > I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed load
> balance strategies on queues aren't coming through versioned flows in NiFi
> Registry. Here are the steps I am performing:
> >
> > 1. Have existing flow running on latest flow version in Dev and Cert
> (flow has already been developed and in version control)
> > 2. Add load balance strategy to queue in Dev flow
> > 3. NiFi shows local changes and I commit the changes to NiFi Registry
> > 4. Cert NIFi shows new flow version
> > 5. Pull latest version down to Cert. The load balance strategy for the
> queue doesn't come with it. I then have to edit flow on Cert to add the
> load balance strategy for the queue which causes NiFi to see local changes
> which I then have to commit to Registry again but this time from Cert.
> >
> > I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194 which
> made me think I shouldn't be experiencing the behavior I am seeing.
> >
> > Thanks,
> > Chad
>

Versioned flows not maintaining Load Balance Strategy

[Chad Woodhead]

I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed load
balance strategies on queues aren't coming through versioned flows in NiFi
Registry. Here are the steps I am performing:

1. Have existing flow running on latest flow version in Dev and Cert (flow
has already been developed and in version control)
2. Add load balance strategy to queue in Dev flow
3. NiFi shows local changes and I commit the changes to NiFi Registry
4. Cert NIFi shows new flow version
5. Pull latest version down to Cert. The load balance strategy for the
queue doesn't come with it. I then have to edit flow on Cert to add the
load balance strategy for the queue which causes NiFi to see local changes
which I then have to commit to Registry again but this time from Cert.

I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194 which
made me think I shouldn't be experiencing the behavior I am seeing.

Thanks,
Chad

Re: Automate NiFi Ranger Policies

[Chad Woodhead]

Kevin,

Thanks for the high level thought process. Seems like a feasible solution.
Do you know if I would be able to get the user who created the "Application
PG" to add them to the Ranger policy so they don't lose access to their own
application? Does NiFi keep that information?

Thanks,
Chad

On Fri, Feb 15, 2019 at 2:02 PM Kevin Doran  wrote:

> Hi Chad,
>
> I've never done this, but if I were to go about it I would create a
> script / cron job to poll the NiFi REST API [1] periodically, and upon
> detection of a new "Application PG", create the corresponding policies
> in Ranger via its REST API [2].
>
> You'll have to create service accounts in both NiFI and Ranger for
> this script to run as and authenticate to both REST APIs, so you'll
> need a secure server to run it from that (1) has access to both NiFi
> and Ranger services and (2) has a way of restricting access to that
> management server, or at least the service account credentials that
> are stored on the server. And of course I would take care when
> automating access policy creation in any service!
>
> Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
> REST API to poll for process groups. I'm not sure if a similar tool
> exists on the Ranger side, although they do have a published Swagger
> spec of their REST API [5][6], so generating something similar to
> NiPyApi using swagger-codegen [7] might be possible. Then again you
> only need to authenticate and access a few endpoints, so any scripting
> language with a decent HTTP client library should be sufficient for
> this type of thing.
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> [2] https://ranger.apache.org/apidocs/index.html
> [3] https://pypi.org/project/nipyapi/
> [4] https://github.com/Chaffelson/nipyapi
> [5] https://ranger.apache.org/apidocs/ui/swagger.json
> [6] https://ranger.apache.org/apidocs/ui/index.html
> [7] https://swagger.io/tools/swagger-codegen/
>
> Hope this helps,
> Kevin
>
> On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodh...@gmail.com)
> wrote:
> > We use Ranger with NiFi for security and we are looking to automate the
> > creation of our Ranger policies.
> >
> > The way we organize our flows is like this:
> > NiFi Root Canvas > Ingest Channel PG > Application PG
> >
> > We create 3 Ranger Policies per Application PG:
> > -/process-groups/
> > -/data/process-groups/
> > -/provenance-data/process-groups/
> >
> > Admins create the Ingest Channel PGs and developers create the
> Application
> > PGs. We were thinking of automating any time a new Application PG is
> > created inside any of the Ingest Channel PGs, create the 3 corresponding
> > Ranger policies. Was curious to see if anyone else has implemented
> anything
> > like this and if so, any tips/suggestions of how to do it?
> >
> > Thanks,
> > Chad
> >
>

Automate NiFi Ranger Policies

[Chad Woodhead]

We use Ranger with NiFi for security and we are looking to automate the
creation of our Ranger policies.

The way we organize our flows is like this:
NiFi Root Canvas > Ingest Channel PG > Application PG

We create 3 Ranger Policies per Application PG:
-/process-groups/
-/data/process-groups/
-/provenance-data/process-groups/

Admins create the Ingest Channel PGs and developers create the Application
PGs. We were thinking of automating any time a new Application PG is
created inside any of the Ingest Channel PGs, create the 3 corresponding
Ranger policies. Was curious to see if anyone else has implemented anything
like this and if so, any tips/suggestions of how to do it?

Thanks,
Chad

Re: Failed to read TOC File

[Chad Woodhead]

Thanks Mark. I really appreciate the help. I'll take a look at these in my
different clusters.

-Chad

On Wed, Feb 13, 2019 at 3:09 PM Mark Payne  wrote:

> Depending on the size of the nodes, you may want to also increase the
> number of indexing threads
> ("nifi.provenance.repository.index.threads"). You may want to also
> increase the amount of time to store provenance
> from 24 hours to something like 36 months... for most people just limiting
> based on size is enough. The time is really
> only useful if you are worried about it from a compliance standpoint, such
> that you are only allowed to store that sort
> of data for a limited amount of time. (Maybe you can only store PII for 24
> hours, for instance, and PII is included in your
> attributes).
>
> The value for "nifi.bored.yield.duration" can probably be scaled back from
> "10 millis" to something like "1 millis". This comes
> into play when a processor's incoming queue is empty or outgoing queue is
> full, for instance. It will wait approximately 10 miliseconds
> before checking if the issue has resolved. This results in lower CPU
> usage, but it also leads to "artificial latency." For a production
> server, "1 millis" is probably just fine.
>
> You may also want to consider changing the values of
> "nifi.content.repository.archive.max.retention.period" and
> "nifi.content.repository.archive.max.usage.percentage"
> When you store the provenance data, it's often helpful to be able to view
> the data as it was at that point in the flow. These properties control how
> long you keep around this data after you've finished processing it, so
> that you can go back and look at it for debugging purposes, etc.
>
> These are probably the most critical things, in terms of performance and
> utilization.
>
> Thanks
> -Mark
>
>
> On Feb 13, 2019, at 2:31 PM, Chad Woodhead  wrote:
>
> Mark,
>
> That must be it! I have "nifi.provenance.repository.max.storage.size" = 1
> GB. Just bumped that to 200 GB like you suggested and I can see provenance
> again. I've always wondered why my provenance partitions never got very
> large!
>
> While we're on the subject, are there other settings like this that based
> on their default values I could be under-utilizing the resources (storage,
> mem, CPU, etc.) I have on my servers dedicated to NiFi?
>
> -Chad
>
> On Wed, Feb 13, 2019 at 1:48 PM Mark Payne  wrote:
>
>> Hey Chad,
>>
>> What do you have for the value of the
>> "nifi.provenance.repository.max.storage.size" property?
>> We will often see this if the value is very small (the default is 1 GB,
>> which is very small) and the volume
>> of data is reasonably high.
>>
>> The way that the repo works, it writes to one file for a while, until
>> that file reaches 100 MB or up to 30 seconds,
>> by default (configured via "nifi.provenance.repository.rollover.size" and
>> "nifi.provenance.repository.rollover.time").
>> At that point, it rolls over to writing to a new file and adds the
>> now-completed file to a queue. A background thread
>> is then responsible for compressing that completed file.
>>
>> What can happen, though, if the max storage space is small is that the
>> data can actually be aged off from the repository
>> before that background task attempts to compress it. That can result in
>> either a FileNotFoundException or an EOFException
>> when trying to read the TOC file (depending on the timing of when the
>> age-off happens). It could potentially occur on the
>> .prov file, in addition to, or instead of the .toc file.
>>
>> So generally, the solution is to increase the max storage size. It looks
>> like you have 130 GB on each partition and 2 partitions per
>> node, so 260 GB total per node that you can use for provenance. So I
>> would set the max storage size to something like "200 GB".
>> Since it is a soft limit and it may use more disk space than that
>> temporarily before shrinking back down, you'll want to give it a little
>> bit of wiggle room.
>>
>> Thanks
>> -Mark
>>
>>
>> On Feb 13, 2019, at 1:31 PM, Chad Woodhead 
>> wrote:
>>
>> Hey Joe,
>>
>> Yes nifi.provenance.repository.implementation=
>> org.apache.nifi.provenance.WriteAheadProvenanceRepository
>>
>> Disk space is fine as well. I have dedicated mounts for provenance (as
>> well all the repos have their own dedicated mounts):
>>
>> nifi.provenance.repository.dir.default=
>> /data/disk5/nifi/provenance_repository
>> 

Re: Failed to read TOC File

[Chad Woodhead]

Mark,

That must be it! I have "nifi.provenance.repository.max.storage.size" = 1
GB. Just bumped that to 200 GB like you suggested and I can see provenance
again. I've always wondered why my provenance partitions never got very
large!

While we're on the subject, are there other settings like this that based
on their default values I could be under-utilizing the resources (storage,
mem, CPU, etc.) I have on my servers dedicated to NiFi?

-Chad

On Wed, Feb 13, 2019 at 1:48 PM Mark Payne  wrote:

> Hey Chad,
>
> What do you have for the value of the
> "nifi.provenance.repository.max.storage.size" property?
> We will often see this if the value is very small (the default is 1 GB,
> which is very small) and the volume
> of data is reasonably high.
>
> The way that the repo works, it writes to one file for a while, until that
> file reaches 100 MB or up to 30 seconds,
> by default (configured via "nifi.provenance.repository.rollover.size" and
> "nifi.provenance.repository.rollover.time").
> At that point, it rolls over to writing to a new file and adds the
> now-completed file to a queue. A background thread
> is then responsible for compressing that completed file.
>
> What can happen, though, if the max storage space is small is that the
> data can actually be aged off from the repository
> before that background task attempts to compress it. That can result in
> either a FileNotFoundException or an EOFException
> when trying to read the TOC file (depending on the timing of when the
> age-off happens). It could potentially occur on the
> .prov file, in addition to, or instead of the .toc file.
>
> So generally, the solution is to increase the max storage size. It looks
> like you have 130 GB on each partition and 2 partitions per
> node, so 260 GB total per node that you can use for provenance. So I would
> set the max storage size to something like "200 GB".
> Since it is a soft limit and it may use more disk space than that
> temporarily before shrinking back down, you'll want to give it a little
> bit of wiggle room.
>
> Thanks
> -Mark
>
>
> On Feb 13, 2019, at 1:31 PM, Chad Woodhead  wrote:
>
> Hey Joe,
>
> Yes nifi.provenance.repository.implementation=
> org.apache.nifi.provenance.WriteAheadProvenanceRepository
>
> Disk space is fine as well. I have dedicated mounts for provenance (as
> well all the repos have their own dedicated mounts):
>
> nifi.provenance.repository.dir.default=
> /data/disk5/nifi/provenance_repository
> nifi.provenance.repository.directory.provenance2=
> /data/disk6/nifi/provenance_repository
>
> Both of these mounts have plenty of space and are only 1% full and have
> never become close to being filled up.
>
> 
>
> -Chad
>
> On Wed, Feb 13, 2019 at 1:06 PM Joe Witt  wrote:
>
>> Chad,
>>
>> In your conf/nifi.properties please see what the implementation is for
>> your provenance repository. This specied on
>>
>>
>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
>>
>> Is that what you have?
>>
>> The above error I believe could occur if the location where provenance is
>> being written runs out of disk space.  It is important to ensure that prov
>> is sized and on a partition alone where this wont happen.  This is also
>> true for the flow file repo.  The content repo is more resilient to this by
>> design but still you want all three repo areas on their own partitions as
>> per best practices.
>>
>> Thanks
>> Joe
>>
>> On Wed, Feb 13, 2019 at 1:03 PM Chad Woodhead 
>> wrote:
>>
>>> I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and
>>> I am seeing the following error in my logs a lot and I can't view any
>>> provenance data in the UI:
>>>
>>> 2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1]
>>> o.a.n.p.s.EventFileCompressor Failed to read TOC File
>>> /data/disk5/nifi/provenance_repository/toc/158994812.toc
>>> java.io.EOFException: null
>>> at
>>> org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
>>> at
>>> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Any ideas on what could be going on?
>>>
>>> -Chad
>>>
>>
>

Failed to read TOC File

[Chad Woodhead]

I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and I
am seeing the following error in my logs a lot and I can't view any
provenance data in the UI:

2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1]
o.a.n.p.s.EventFileCompressor Failed to read TOC File
/data/disk5/nifi/provenance_repository/toc/158994812.toc
java.io.EOFException: null
at
org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Any ideas on what could be going on?

-Chad

Re: AmbariReportingTask help

[Chad Woodhead]

Hey Mark,

I see. That makes sense why I didn’t see metrics for flowfiles coming from 
GenerateFlowFile processor.

When using custom NARs to consume from external sources, is there any code that 
needs to be added to the NAR so the processor’s metrics are reported in this 
category or does NiFi handle that automatically?

-Chad


> On Feb 13, 2019, at 9:44 AM, Mark Payne  wrote:
> 
> Chad,
> 
> It represents any FlowFile that was received from an external source. This 
> could be via
> Site-to-Site or could be from something like GetHTTP, FetchSFTP, etc. It 
> correlates
> to any RECEIVE or FETCH provenance events.
> 
> If you go to the Summary table (menu in the top right / Summary) and then go 
> to the
> Process Groups tab, you can see the Sent/Received metrics over the past 5 
> minutes there.
> 
> Thanks
> -Mark
> 
> 
>> On Feb 13, 2019, at 9:32 AM, Chad Woodhead  wrote:
>> 
>> I was looking for some clarification on the AmbariReportingTask, and 
>> specifically the metrics FlowFilesReceivedLast5Minutes and 
>> BytesReceivedLast5Minutes. 
>> 
>> Looking at the configuration for AmbariReportingTask, it has the property 
>> ‘Process Group ID’ and explains that if left blank, the root process group 
>> is used and global metrics are sent. (I have ‘Process Group ID’ set to blank 
>> in my AmbariReportingTask)  So for global metrics on the root process group, 
>> what does it use as a metric for FlowFilesReceived? All processors or just 
>> the first processor in a flow considered for this metric? Is there a way to 
>> see these metrics on the root process group in NiFi similar to ‘Status 
>> History’ on a component?
>> 
>> Thanks,
>> Chad
> 

AmbariReportingTask help

[Chad Woodhead]

I was looking for some clarification on the AmbariReportingTask, and 
specifically the metrics FlowFilesReceivedLast5Minutes and 
BytesReceivedLast5Minutes. 

Looking at the configuration for AmbariReportingTask, it has the property 
‘Process Group ID’ and explains that if left blank, the root process group is 
used and global metrics are sent. (I have ‘Process Group ID’ set to blank in my 
AmbariReportingTask)  So for global metrics on the root process group, what 
does it use as a metric for FlowFilesReceived? All processors or just the first 
processor in a flow considered for this metric? Is there a way to see these 
metrics on the root process group in NiFi similar to ‘Status History’ on a 
component?

Thanks,
Chad

Re: process group name reverts back to initial value if I do a nifi registry "Change version"

[Chad Woodhead]

Bryan/Josef,

Just wanted to let you know that I just tested this with NiFi 1.8.0 and NiFi 
Registry 0.3.0 and I experience the same behavior as Josef.

-Chad

> On Jan 8, 2019, at 12:58 PM, Bryan Bende  wrote:
> 
> I will keep trying, but I haven't been able to reproduce using NiFi
> 1.8.0 and registry 0.2.0.
> 
> I must be missing something.
> 
> On Tue, Jan 8, 2019 at 11:50 AM  wrote:
>> 
>> I've tried it now on another secured cluster exactly like you did:
>> 
>>1) Create PG "A" and save to registry
>>2) Import PG "A" from registry and rename to "B"
>>3) Add new processor (Execute Script) to original PG "A" and save to 
>> registry
>>4) Change version on PG "B" to new version
>> 
>> Problem still there... after changing to new version on "B" the name changed 
>> back to "A"
>> 
>> 
>> 
>> On 08.01.19, 17:40, "Zahner Josef, GSB-LR-TRW-LI" 
>>  wrote:
>> 
>>Hi Bryan
>> 
>>In my case it happens all the time, doesn't matter what kind of change. 
>> On my first test below I've changed a variable on a processor inside the PG 
>> and the second time (a few seconds ago) I've added a connection to my 
>> "Execute Script" processor. All the time my second PG with the new name 
>> changed back to the initial name... Even if I just click "Change version" 
>> and select another one than the current, my second PG changes the name back 
>> to the initial value.
>> 
>>Btw. we use NiFi Registry 0.2.0.
>> 
>>Cheers Josef
>> 
>> 
>> 
>> 
>> 
>>On 08.01.19, 17:23, "Bryan Bende"  wrote:
>> 
>>Hi Josef,
>> 
>>That sounds like a possible bug. I think the PG name is supposed to
>>remain unchanged.
>> 
>>I wasn't able to reproduce this though... in step 5 when you change
>>the "abc" group, what type of change are you making?
>> 
>>I did the following...
>> 
>>1) Create PG "A" and save to registry
>>2) Import PG "A" from registry and rename to "B"
>>3) Add new processor to original PG "A" and save to registry
>>4) Change version on PG "B" to new version
>> 
>>PG "B" is still named "B" at this point.
>> 
>>On Tue, Jan 8, 2019 at 10:26 AM  wrote:
>>> 
>>> Hi guys
>>> 
>>> 
>>> 
>>> I’ve faced again an (at least for me) unexpected behavior of NiFi 1.8.0 
>>> together with NiFi Registry.
>>> 
>>> 
>>> 
>>> Following use case:
>>> 
>>> 
>>> 
>>> Create a process group with name “abc” and add some processors to the pg
>>> Commit the pg to the NiFi registry
>>> Create a new pg and import the pg from step 1 from the registry
>>> Change the name of the new pg to “def” instead of “abc” – so far so good, 
>>> no change from registry point of view
>>> Change the original pg “abc” from step 1 and commit the change to the 
>>> registry
>>> Now we have change to the newest version for the pg “def” from step 4, as 
>>> it isn’t anymore up to date – but now in my case as soon as I’m changing 
>>> the version, the pg name gets changed back to “abc”. This happens all the 
>>> time if I change the version on a pg which has another name than on the 
>>> commit
>>> 
>>> 
>>> 
>>> Any comments on this? We use the NiFi registry as well as templating 
>>> infrastructure, means we have several times the same pg but with different 
>>> variables and names on the same NiFi canvas. But with the actual behavior 
>>> this is very inconvenient… we have to memorize the name before we do the 
>>> “Change version” and then after execution we have to set it again.
>>> 
>>> 
>>> 
>>> Cheers Josef
>> 
>> 
>> 
>> 

Re: Cron-schedule not working

[Chad Woodhead]

Hi Asanka,

Yes you are correct, my mistake. NiFi uses the Quartz Cron and not the Linux 
cron.

Every day at midnight is: 0 0 0 * * ?

My apologies.

-Chad

> On Jan 1, 2019, at 1:55 AM, Asanka Sanjaya  wrote:
> 
> Hi Chad,
> 
> The nifi docs say that the expression should start with seconds and not with 
> minutes.  https://nifi.apache.org/docs/nifi-docs/html/user-guide.html
> 
> 
> 
> This is the example that they have given: 
> The string 0 0 13 * * ? indicates that you want to schedule the processor to 
> run at 1:00 PM every day.
> 
> It seems like nifi cron expressions are quite different.
> 
> 
>> On Tue, Jan 1, 2019 at 11:58 AM Asanka Sanjaya  wrote:
>> Thanks Chad! 
>> 
>>> On Mon, Dec 31, 2018 at 6:22 PM Chad Woodhead  
>>> wrote:
>>> Hi Asanka,
>>> 
>>> The cron expression for every night at midnight is: 0 0 * * *
>>> 
>>> 0 0 0 * * is not a valid cron expression as the 3rd field (day of month) 
>>> cannot be a 0. 
>>> 
>>> Here is an online cron editor that can help build your cron expressions: 
>>> https://crontab.guru/ 
>>> 
>>> Also here is simple cron tester to see the next n iterations of your cron 
>>> expression: http://cron.schlitt.info/
>>> 
>>> -Chad
>>> 
>>>> On Dec 31, 2018, at 2:04 AM, Asanka Sanjaya  wrote:
>>>> 
>>>> Hello,
>>>> 
>>>> I use a cron-driven processor to start the workflow and the expression is:
>>>> 
>>>> 0 0 0 * * ?
>>>> 
>>>> I expect the processor to run each day at 00:00 hours. But it runs each 
>>>> hour instead. 
>>>> 
>>>> Processor configuration:
>>>> 
>>>> 
>>>> 
>>>> Flow:
>>>> 
>>>> 
>>>> 
>>>> What could be the possible reasons for this?
>>>> 
>>>> 
>>>> -- 
>>>> Thanks,
>>>> Asanka Sanjaya Herath
>>>> Senior Software Engineer | Zone24x7
>>> 
>> 
>> 
>> -- 
>> Thanks,
>> Asanka Sanjaya Herath
>> Senior Software Engineer | Zone24x7
> 
> 
> -- 
> Thanks,
> Asanka Sanjaya Herath
> Senior Software Engineer | Zone24x7

Re: Cron-schedule not working

[Chad Woodhead]

Hi Asanka,

The cron expression for every night at midnight is: 0 0 * * *

0 0 0 * * is not a valid cron expression as the 3rd field (day of month) cannot 
be a 0. 

Here is an online cron editor that can help build your cron expressions: 
https://crontab.guru/  

Also here is simple cron tester to see the next n iterations of your cron 
expression: http://cron.schlitt.info/ 

-Chad

> On Dec 31, 2018, at 2:04 AM, Asanka Sanjaya  wrote:
> 
> Hello,
> 
> I use a cron-driven processor to start the workflow and the expression is:
> 
> 0 0 0 * * ?
> 
> I expect the processor to run each day at 00:00 hours. But it runs each hour 
> instead. 
> 
> Processor configuration:
> 
> 
> 
> Flow:
> 
> 
> 
> What could be the possible reasons for this?
> 
> 
> -- 
> Thanks,
> Asanka Sanjaya Herath
> Senior Software Engineer | Zone24x7