Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)

[Bryan Bende]

+1 binding

Ran through release helper
Ran simple flow
Saved flows to registry, verified DB persistence is fixed

Thanks David!

On Fri, Jan 26, 2024 at 1:33 PM Robert Fellows  wrote:
>
> +1 non-binding
>
> verified hashes
> build with java 21
> ran simple flow
> built with new UI
>
>
> On Fri, Jan 26, 2024 at 12:11 PM Nissim Shiman 
> wrote:
>
> >  +1 (non-binding)
> >
> > verified source release sha256/512 checksums
> >
> > successfully ran build using:
> > Apache Maven 3.9.6
> > Java 21 2023-09-19 LTS
> > linux kernel 3.10.0-1160
> >
> > Ran various simple flows successfully.  Migrated registry from 1.24 and
> > 2.0.0-M1 to this RC successfully.  (i.e. and tested importing PGs,
> > committing new versions of PGs to registry successfully, both from nifi and
> > filesystem).  Verified flows were persisted in registry after registry
> > restart as well.
> >
> > Noticed non-blocking issue where reporting tasks' link to referenced
> > controller service is not completely working.  Created NIFI-12678 for it.
> >
> > Tested:
> > NIFI-11389 Controller Services's link to referencing Controller Services
> > components not always working
> > NIFI-12387 Flow Configuration History can record a Comments change for
> > Controller Service when no changes have been made
> >
> > NIFI-12394 when importing versioned flow with component that migrates
> > properties, controller service reference is invalid
> > NIFI-12666 Correct NiFi Registry Data Source Configuration
> >
> > Thank you David and team for the upcoming release!
> >
> > Nissim Shiman
> >
> > On Friday, January 26, 2024 at 02:30:45 AM UTC, David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> >  Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 2.0.0-M2.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on and the convenience binaries are available
> > on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1264
> >
> > Git Tag: nifi-2.0.0-M2-RC4
> > Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> > GitHub Commit Link:
> >
> > https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> >
> > Checksums of nifi-2.0.0-M2-source-release.zip
> >
> > SHA256: 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b
> > SHA512:
> > e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 214
> >
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353861
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [] +1 Release this package as nifi-2.0.0-M2
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>
>
>
> --
> ---
> Rob Fellows

Re: [DISCUSS] Validator hook for Registry events

[Bryan Bende]

Good point, since registry client is an extension point, using the
rules on NiFi side allows it to work for any registry client.

On Wed, Jan 17, 2024 at 11:49 AM Pierre Villard
 wrote:
>
> Yeah so the more I think about it the more we should probably couple this
> with the new rules engine feature of NiFi 2.0. Users could define their
> rules there and then it could just be a property in NiFi saying whether or
> not it should prevent someone from committing a new version if one of the
> required rules is raising a violation in the process group being committed.
> This way this would work with any registry implementation, not just the
> NiFi Registry (I'm saying this in case we ever get to a pure Git-based
> implementation of the Registry endpoint).
>
> Le mer. 17 janv. 2024 à 20:33, Bryan Bende  a écrit :
>
> > Michael,
> >
> > I think this validation idea came from the absence of a pull request
> > review model. Meaning, the rules being discussed here would check for
> > the things someone would most likely look for in a review of the flow
> > changes to decide if the flow is good. Obviously it wouldn't be able
> > to do everything a person would, but it could be better than nothing.
> > As far as I know, there isn't any active work around building a review
> > model.
> >
> > -Bryan
> >
> > On Wed, Jan 17, 2024 at 11:18 AM Michael Hogue
> >  wrote:
> > >
> > > Could this be accomplished through git webhooks once changes are
> > committed
> > > through Registry and CI pipelines to perform whatever validation you
> > wish?
> > >
> > > The glaring weakness in this approach is that the changes have already
> > been
> > > committed. This makes me wonder if we can improve the "flow development
> > > lifecycle" to enable teams to review flow changes before they're
> > > committed, much like a pull request. Is this on the roadmap for Registry?
> > >
> > > Mike
> > >
> > > On Wed, Jan 17, 2024 at 4:14 PM Bryan Bende  wrote:
> > >
> > > > Thanks guys, that makes sense in the context of the review model.
> > > >
> > > > Obviously some kind of general pre-event hook is the most flexible,
> > > > but I would say we should also consider whether we really want to be
> > > > calling out to arbitrary scripts during the main API requests, as well
> > > > as consider what someone would have to do to implement a scripted
> > > > rule. The current events are all metadata related, they don't have the
> > > > content of the snapshots, so in this case we'd have to pass the entire
> > > > json string of a snapshot as an arg to the script and then the script
> > > > has to parse through it looking for fields to validate/check. I wonder
> > > > if we should consider defining the types of rules ourselves and
> > > > allowing some kind of rules config file to be provided. This way the
> > > > rules are loaded during start up and applied in Java code by our
> > > > framework code. Downside is it is less flexible in that we probably
> > > > won't be able to know every rule someone wants ahead of time. I guess
> > > > the other options are to consider whether tagging or nifi side rules
> > > > are better suited to solve this problem.
> > > >
> > > > On Wed, Jan 17, 2024 at 10:48 AM Simon Bence  > >
> > > > wrote:
> > > > >
> > > > > HI,
> > > > >
> > > > > I think, both of you are correct, my initial example of the possible
> > > > usage was not the best. Having a control over committing versions for
> > flows
> > > > is maybe the most important benefit we can gain. Contrary to naming
> > > > conventions this can be a more complex endeavour. So I suggest moving
> > on
> > > > with this example instead of the original one.
> > > > >
> > > > > Regards,
> > > > > Bence
> > > > >
> > > > > > On 2024. Jan 17., at 16:02, Pierre Villard <
> > > > pierre.villard...@gmail.com> wrote:
> > > > > >
> > > > > > I guess the issue relates to some recurrent discussions around the
> > > > lack of
> > > > > > mechanism for a Pull Request kind of model for "approving" a
> > versioned
> > > > > > flow. An alternative we've been discussing for a long time is the
> > > > ability
> > > > > > to set tags to versions 

Re: [DISCUSS] Validator hook for Registry events

[Bryan Bende]

Michael,

I think this validation idea came from the absence of a pull request
review model. Meaning, the rules being discussed here would check for
the things someone would most likely look for in a review of the flow
changes to decide if the flow is good. Obviously it wouldn't be able
to do everything a person would, but it could be better than nothing.
As far as I know, there isn't any active work around building a review
model.

-Bryan

On Wed, Jan 17, 2024 at 11:18 AM Michael Hogue
 wrote:
>
> Could this be accomplished through git webhooks once changes are committed
> through Registry and CI pipelines to perform whatever validation you wish?
>
> The glaring weakness in this approach is that the changes have already been
> committed. This makes me wonder if we can improve the "flow development
> lifecycle" to enable teams to review flow changes before they're
> committed, much like a pull request. Is this on the roadmap for Registry?
>
> Mike
>
> On Wed, Jan 17, 2024 at 4:14 PM Bryan Bende  wrote:
>
> > Thanks guys, that makes sense in the context of the review model.
> >
> > Obviously some kind of general pre-event hook is the most flexible,
> > but I would say we should also consider whether we really want to be
> > calling out to arbitrary scripts during the main API requests, as well
> > as consider what someone would have to do to implement a scripted
> > rule. The current events are all metadata related, they don't have the
> > content of the snapshots, so in this case we'd have to pass the entire
> > json string of a snapshot as an arg to the script and then the script
> > has to parse through it looking for fields to validate/check. I wonder
> > if we should consider defining the types of rules ourselves and
> > allowing some kind of rules config file to be provided. This way the
> > rules are loaded during start up and applied in Java code by our
> > framework code. Downside is it is less flexible in that we probably
> > won't be able to know every rule someone wants ahead of time. I guess
> > the other options are to consider whether tagging or nifi side rules
> > are better suited to solve this problem.
> >
> > On Wed, Jan 17, 2024 at 10:48 AM Simon Bence 
> > wrote:
> > >
> > > HI,
> > >
> > > I think, both of you are correct, my initial example of the possible
> > usage was not the best. Having a control over committing versions for flows
> > is maybe the most important benefit we can gain. Contrary to naming
> > conventions this can be a more complex endeavour. So I suggest moving on
> > with this example instead of the original one.
> > >
> > > Regards,
> > > Bence
> > >
> > > > On 2024. Jan 17., at 16:02, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > I guess the issue relates to some recurrent discussions around the
> > lack of
> > > > mechanism for a Pull Request kind of model for "approving" a versioned
> > > > flow. An alternative we've been discussing for a long time is the
> > ability
> > > > to set tags to versions so that a versioned flow could be reviewed
> > (diff
> > > > with previous version(s)) and if approved a tag would be applied and
> > this
> > > > tag would be used to potentially trigger an automatic deployment.
> > > >
> > > > The proposal is not helping with the "PR model" but would allow
> > someone to
> > > > control what is versioned. A user may have the permissions to commit a
> > new
> > > > version but one may not want to accept a flow where a processor is
> > > > configured with 50 concurrent tasks (for example). That would get
> > deployed
> > > > in prod and potentially impact existing deployed flows. I guess one
> > could
> > > > see this approach as a "checkstyle" plugin to allow or not the commit
> > of a
> > > > new version.
> > > >
> > > > Le mer. 17 janv. 2024 à 18:56, Bryan Bende  a écrit
> > :
> > > >
> > > >> I think before we embed something into synchronous API requests we
> > > >> should try to define a few more real uses besides just a bucket naming
> > > >> pattern to see if we really need to add something like this. For
> > > >> bucket naming pattern, it could just be a simple property in
> > > >> nifi-registry.properties with a regex to check names against. If we
> > > >> did add some kind of pluggable validator, I don't think these
> &

Re: [DISCUSS] Validator hook for Registry events

[Bryan Bende]

Thanks guys, that makes sense in the context of the review model.

Obviously some kind of general pre-event hook is the most flexible,
but I would say we should also consider whether we really want to be
calling out to arbitrary scripts during the main API requests, as well
as consider what someone would have to do to implement a scripted
rule. The current events are all metadata related, they don't have the
content of the snapshots, so in this case we'd have to pass the entire
json string of a snapshot as an arg to the script and then the script
has to parse through it looking for fields to validate/check. I wonder
if we should consider defining the types of rules ourselves and
allowing some kind of rules config file to be provided. This way the
rules are loaded during start up and applied in Java code by our
framework code. Downside is it is less flexible in that we probably
won't be able to know every rule someone wants ahead of time. I guess
the other options are to consider whether tagging or nifi side rules
are better suited to solve this problem.

On Wed, Jan 17, 2024 at 10:48 AM Simon Bence  wrote:
>
> HI,
>
> I think, both of you are correct, my initial example of the possible usage 
> was not the best. Having a control over committing versions for flows is 
> maybe the most important benefit we can gain. Contrary to naming conventions 
> this can be a more complex endeavour. So I suggest moving on with this 
> example instead of the original one.
>
> Regards,
> Bence
>
> > On 2024. Jan 17., at 16:02, Pierre Villard  
> > wrote:
> >
> > I guess the issue relates to some recurrent discussions around the lack of
> > mechanism for a Pull Request kind of model for "approving" a versioned
> > flow. An alternative we've been discussing for a long time is the ability
> > to set tags to versions so that a versioned flow could be reviewed (diff
> > with previous version(s)) and if approved a tag would be applied and this
> > tag would be used to potentially trigger an automatic deployment.
> >
> > The proposal is not helping with the "PR model" but would allow someone to
> > control what is versioned. A user may have the permissions to commit a new
> > version but one may not want to accept a flow where a processor is
> > configured with 50 concurrent tasks (for example). That would get deployed
> > in prod and potentially impact existing deployed flows. I guess one could
> > see this approach as a "checkstyle" plugin to allow or not the commit of a
> > new version.
> >
> > Le mer. 17 janv. 2024 à 18:56, Bryan Bende  a écrit :
> >
> >> I think before we embed something into synchronous API requests we
> >> should try to define a few more real uses besides just a bucket naming
> >> pattern to see if we really need to add something like this. For
> >> bucket naming pattern, it could just be a simple property in
> >> nifi-registry.properties with a regex to check names against. If we
> >> did add some kind of pluggable validator, I don't think these
> >> validators should be making authorization decisions, it should be
> >> strictly for rules about the content being version controlled. If we
> >> are worried about a user version controlling something that interferes
> >> with a CI/CD pipeline, that seems to imply our current authorization
> >> policy model needs improvement, or it is not being used correctly
> >> (i.e. why does a user have write to a bucket they shouldn't be writing
> >> to?).
> >>
> >> On Wed, Jan 17, 2024 at 8:53 AM Pierre Villard
> >>  wrote:
> >>>
> >>> I do think this would be a good idea. That would provide users of the
> >> NiFi
> >>> Registry a way to implement custom conditions on what can be versioned or
> >>> not based on their requirements. Right now if one has write permissions
> >> for
> >>> a bucket/flow, you could commit anything as a new version which could
> >>> potentially be an issue when using CI/CD pipelines to automate
> >> deployments.
> >>>
> >>> As a side-note, another improvement could be to prevent a flow from being
> >>> versioned in the registry if a required rule is violated (I'm talking
> >> about
> >>> the new feature coming with NiFi 2.0). Just a thought for an additional
> >>> improvement.
> >>>
> >>> Overall I do think that giving options to users for better controlling
> >> what
> >>> is being versioned is good.
> >>>
> >>> Le mer. 17 janv. 2024 à 17:13, Simon Bence  a
> >>> écrit :

Re: [DISCUSS] Validator hook for Registry events

[Bryan Bende]

I think before we embed something into synchronous API requests we
should try to define a few more real uses besides just a bucket naming
pattern to see if we really need to add something like this. For
bucket naming pattern, it could just be a simple property in
nifi-registry.properties with a regex to check names against. If we
did add some kind of pluggable validator, I don't think these
validators should be making authorization decisions, it should be
strictly for rules about the content being version controlled. If we
are worried about a user version controlling something that interferes
with a CI/CD pipeline, that seems to imply our current authorization
policy model needs improvement, or it is not being used correctly
(i.e. why does a user have write to a bucket they shouldn't be writing
to?).

On Wed, Jan 17, 2024 at 8:53 AM Pierre Villard
 wrote:
>
> I do think this would be a good idea. That would provide users of the NiFi
> Registry a way to implement custom conditions on what can be versioned or
> not based on their requirements. Right now if one has write permissions for
> a bucket/flow, you could commit anything as a new version which could
> potentially be an issue when using CI/CD pipelines to automate deployments.
>
> As a side-note, another improvement could be to prevent a flow from being
> versioned in the registry if a required rule is violated (I'm talking about
> the new feature coming with NiFi 2.0). Just a thought for an additional
> improvement.
>
> Overall I do think that giving options to users for better controlling what
> is being versioned is good.
>
> Le mer. 17 janv. 2024 à 17:13, Simon Bence  a
> écrit :
>
> > Hi,
> >
> > I recently found the need for custom validation to maintain NiFi Registry
> > content. This includes checks such as enforcing naming conventions when
> > creating a Bucket and similar usage specific cases. While exploring the
> > Registry's codebase, I came across the EventHookProvider, which aligns with
> > a similar concept. However, it does not cover the case here due to its
> > asynchronous nature and being a "post-event" activity.
> >
> > Although the EventHookProvider is not suitable for this specific need, I
> > find the Event construct and the "whitelist" concept pretty overlapping
> > with my objectives. Consequently, I propose the addition of a new type of
> > Provider covering for "pre-event" validation, operating in a manner similar
> > to the EventHookProvider: a call from the request methods to the set of
> > providers but filtered using a whitelist. Similarly to the mentioned
> > provider, I believe an implementation capable of executing scripts (akin to
> > ScriptEventHookProvider) would be a good starting point, to cover a most
> > use cases.
> >
> > I am keen to hear your opinion on this proposal and welcome any further
> > ideas. Thank you for your consideration!
> >
> > PS.: using the "event" term comes from the already existing
> > EventHookProvider. In practice these are the methods of the Registry web
> > API.
> >
> > Regards,
> > Bence Simon

Re: [DISCUSS] State Management improvements for DR scenarios

[Bryan Bende]

erned about a composite state manager provider. It seems prone to
> > getting providers out of sync, as either backing state store could fail,
> > resulting in inconsistent state across regions. This is the primary reason
> > why it seems better to evaluate a resilient storage solution instead of a
> > composite solution.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Dec 11, 2023, 6:23 AM Pierre Villard 
> > wrote:
> >
> > > Hi David and Bryan, thanks for the feedback.
> > >
> > > David,
> > >
> > > Regarding the source and destination. It was just an example. There are
> > > plenty of different use cases you can think of. Of course, this is
> > assuming
> > > that the user of NiFi has a source and destination which are highly
> > > available but this is the user's responsibility. You can take the example
> > > of ListSFTP -> FetchSFTP -> doSomething -> PutS3.
> > >
> > > I definitely agree that having a state manager that supports cross region
> > > replication would be ideal (and IIRC Redis does support this). The
> > approach
> > > of leader/follower makes it easier from a user point of view but I'm
> > > definitely fine considering state manager implementations - for example,
> > in
> > > the past, I suggested a database based implementation (Postgres would be
> > a
> > > good candidate for example).
> > >
> > > When you say it's adding a layer of complexity, it definitely does but
> > only
> > > for users that are looking for options when it comes to DR. In any case,
> > > setting up any piece of software to support DR scenarios requires a lot
> > of
> > > work and is a lot of complexity. However the proposal is not changing
> > > anything to the default behavior and most users would not ever care about
> > > the custom state ID. The goal is to provide ways for a user to support DR
> > > scenarios.
> > >
> > > The big topic is the state ID, right now the coupling between the UUID
> > and
> > > the state is making things extremely complex.
> > >
> > > Bryan,
> > >
> > > Yeah that could be an option I guess. Need to think more about it.
> > >
> > >
> > > Le lun. 11 déc. 2023 à 03:39, Bryan Bende  a écrit :
> > >
> > > > Just a couple of thoughts to add to the discussion...
> > > >
> > > > The concept of the composite state manager really doesn't require any
> > > > framework level changes right? Even if we chose not to provide it out
> > > > of the box, it could still be implemented by anyone in a custom NAR.
> > > >
> > > > For the state id, I do think this is a common problem for anyone
> > > > trying to recreate a flow and pick up where a previous instance of the
> > > > flow left off. I'm not sure if this would make it any simpler, but I
> > > > wonder if something could be done at a process group level when a
> > > > process group is under version control. If there was a unique id that
> > > > could be set at the versioned process group level, essentially an
> > > > instance/deployment id of the flow, then the component state id could
> > > > be the versioned component id + the versioned PG deployment id. This
> > > > could be done for all stateful components in the versioned PG, rather
> > > > than needing to enter it for every stateful component. It would still
> > > > require setting this new id manually, but just moves it a level higher
> > > > from individual components.
> > > >
> > > >
> > > > On Sat, Dec 9, 2023 at 5:48 PM David Handermann
> > > >  wrote:
> > > > >
> > > > > Pierre,
> > > > >
> > > > > Thanks for taking the time to put together the feature proposal with
> > > > > additional background on the related Jira issues. The topic of
> > > > > disaster recovery is an important and challenging one, so it is
> > > > > definitely worth careful consideration.
> > > > >
> > > > > At a high level, I think it is worth considering some additional use
> > > > > case flows, as that would highlight some additional considerations.
> > > > >
> > > > > Taking the ListS3 to PutDatabaseRecord example, it raises some
> > > > > concerns in itself that might be addressed in better ways. For
> > > > > exampl

Re: [DISCUSS] State Management improvements for DR scenarios

[Bryan Bende]

Just a couple of thoughts to add to the discussion...

The concept of the composite state manager really doesn't require any
framework level changes right? Even if we chose not to provide it out
of the box, it could still be implemented by anyone in a custom NAR.

For the state id, I do think this is a common problem for anyone
trying to recreate a flow and pick up where a previous instance of the
flow left off. I'm not sure if this would make it any simpler, but I
wonder if something could be done at a process group level when a
process group is under version control. If there was a unique id that
could be set at the versioned process group level, essentially an
instance/deployment id of the flow, then the component state id could
be the versioned component id + the versioned PG deployment id. This
could be done for all stateful components in the versioned PG, rather
than needing to enter it for every stateful component. It would still
require setting this new id manually, but just moves it a level higher
from individual components.


On Sat, Dec 9, 2023 at 5:48 PM David Handermann
 wrote:
>
> Pierre,
>
> Thanks for taking the time to put together the feature proposal with
> additional background on the related Jira issues. The topic of
> disaster recovery is an important and challenging one, so it is
> definitely worth careful consideration.
>
> At a high level, I think it is worth considering some additional use
> case flows, as that would highlight some additional considerations.
>
> Taking the ListS3 to PutDatabaseRecord example, it raises some
> concerns in itself that might be addressed in better ways. For
> example, instead of relying on localized cluster state, a more
> resilient approach could make use of an event queueing system like
> Amazon SQS or SNS based on S3 Event Notifications. That avoids ListS3
> entirely and provides a more fault-tolerant architecture for tracking
> S3 items to be processed.
>
> The other half of the equation is the destination database. Although
> it is also external to NiFi, the example provided implies that the
> destination supports global redundancy such that communication from
> different regions remains possible in the event of a single region
> failure. That is certainly possible with various storage solutions, it
> just highlights the fact that a true disaster recovery configuration
> requires end-to-end design.
>
> In the initial proposal, the diagrams show regional State Management
> solutions. The concept of a composite state management solution is
> interesting, but it seems to be attempting to make up for the lack of
> a true distributed, resilient, and cross-region state management
> solution. Granted, ZooKeeper and Kubernetes ConfigMap storage may not
> be a good fit for a cross-region solution. However, it seems like it
> would be better to evaluate an optimal cross-region state management
> implementation, as opposed implementing some type of replication or
> leader-follower design in NiFi itself.
>
> To be clear, this is certainly a topic worth considering, but I am not
> confident that the implementation steps outlined in the initial two
> Jira issues will provide a robust or maintainable solution. Supporting
> component-level configuration of a custom state identifier seems prone
> to error, and also requires a lot of manual configuration at the
> individual Processor level. Supporting a composite state management
> could have other benefits, but it also adds a layer of complexity that
> may not even achieve the desired outcome, depending on the
> capabilities of the underlying storage implementations.
>
> With that background, I think it would be worth evaluating alternative
> approaches before moving to any kind of implementation. I'm sure there
> are aspects I have not considered, so I welcome additional perspective
> on the positives and negatives of the proposed solution.
>
> Regards,
> David Handermann
>
> On Fri, Dec 8, 2023 at 8:32 AM Pierre Villard
>  wrote:
> >
> > Team,
> >
> > I just published a feature proposal here:
> > https://cwiki.apache.org/confluence/display/NIFI/State+Management+improvements+for+Disaster+Recovery+scenarios
> >
> > This feature proposal is to provide a more detailed explanation around the
> > two below JIRAs:
> > https://issues.apache.org/jira/browse/NIFI-11776
> > https://issues.apache.org/jira/browse/NIFI-11777
> >
> > I'd love to hear your thoughts before we get started with the actual
> > implementation.
> >
> > Thanks,
> > Pierre

Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)

[Bryan Bende]

+1 binding

Ran through the release helper and a sample flow, thanks Pierre.

On Fri, Nov 17, 2023 at 3:46 PM Joe Witt  wrote:

> +1 binding
>
> Thanks everyone.  Thanks Pierre!
>
> On Fri, Nov 17, 2023 at 1:39 PM Nissim Shiman 
> wrote:
>
> >  +1 (non-binding)
> >
> > verified source release sha256/512 checksums
> >
> > successfully ran build using:
> > Apache Maven 3.9.5
> > Java openjdk version: 1.8.0_382
> > linux kernel 3.10.0-1160
> >
> > Ran various simple flows successfully.  Migrated registry from 1.23.2 to
> > this RC successfully.  (i.e. and tested importing PGs, committing new
> > versions of PGs to registry successfully.)
> >
> > Tested/verified:
> > NIFI-11782 NPE when adding Snippet with label into Process Group
> > NIFI-12084 Logging by process group is not workin on 1.x support branch
> > (thank you to all those who worked on this and its parent ticket,
> > NIFI-3065.  This new granular level of logging is really nice)
> >
> > While testing, noticed/filed non holdup issues:
> > NIFI-12387 Flow Configuration History can record a Comments change for
> > Controller Service when no changes have been made
> > NIFI-12388 Process Group log file suffix can become out of sync with its
> > name when PG was copied from a version controlled PG
> >
> > Thank you Pierre and team for the upcoming release!
> >
> > Nissim Shiman
> >
> >
> > On Friday, November 17, 2023 at 08:10:20 PM UTC, Dan S <
> > dsti...@gmail.com> wrote:
> >
> >  +1 (non-binding)
> >
> >   - Went through the Release Candidate Verification
> >   <
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > >
> >   - Verified signatures and hashes
> >   - Built on CentOS Linux 7
> >   - Java version Oracle 17.0.7
> >   - Maven version  3.9.5
> >   - Executed simple flow to exercise NIFI-11197
> >   
> >   - Confirmed fix for NIFI-12165
> >   
> >   - Confirmed fix for NIFI-11959
> >   
> >
> > Thanks for RM'ing Pierre!
> >
> > On Fri, Nov 17, 2023 at 1:01 PM Marton Szasz  wrote:
> >
> > > +1 (binding)
> > >
> > > Compiled and ran with Java 17 on Ubuntu 22.04, executing a simple flow.
> > >
> > > On Fri, Nov 17, 2023 at 6:59 PM Matt Burgess 
> > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > Ran through release helper, ran NiFi standalone with Java 8, tested
> > > > various flows and components.  Thanks for RM'ing Pierre!
> > > >
> > > > On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard
> > > >  wrote:
> > > > >
> > > > > Team,
> > > > >
> > > > > I am pleased to be calling this vote for the source release of
> Apache
> > > NiFi
> > > > > 1.24.0.
> > > > >
> > > > > Please review the following guide for how to verify a release
> > candidate
> > > > > build:
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > > >
> > > > > The source being voted on the and the convenience binaries are
> > > available on
> > > > > the Apache Distribution Repository:
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
> > > > >
> > > > > The build artifacts are available on the Apache Nexus Repository:
> > > > >
> > > > >
> > https://repository.apache.org/content/repositories/orgapachenifi-1241
> > > > >
> > > > > Git Tag: nifi-1.24.0-RC3
> > > > > Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > > > GitHub Commit Link:
> > > > >
> > >
> >
> https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > > >
> > > > > Checksums of nifi-1.24.0-source-release.zip
> > > > >
> > > > > SHA256:
> > > 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a
> > > > > SHA512:
> > > > >
> > >
> >
> 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8
> > > > >
> > > > > Release artifacts are signed with the following key:
> > > > >
> > > > > https://people.apache.org/keys/committer/pvillard.asc
> > > > >
> > > > > KEYS file is available on the Apache Distribution Repository:
> > > > >
> > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > > >
> > > > > Issues resolved for this version: 280
> > > > >
> > > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353443
> > > > >
> > > > > Release note highlights can be found on the project wiki:
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
> > > > >
> > > > > The vote will be open for 72 hours.
> > > > >
> > > > > Please download the release candidate and evaluate the necessary
> > items
> > > > > including checking hashes, signatures, build from source, and test.
> > > Then
> > > > > please vote:
> > > > >
> > > > > [] +1 Release this package as nifi-1.24.0
> > > > > [] +0 no opinion
> > > > > [] -1 Do 

Re: Default parameter storage

[Bryan Bende]

Parameters are always stored in NiFi itself, encrypted in
flow.xml/flow.json. Parameter providers are a way to easily import
parameters and values from an external source like Vault and other various
secret stores, but they are still ultimately brought into NiFi and stored
locally.

NiFi Registry is not a parameter provider. When a flow is saved to NiFi
registry, it contains a snapshot of what the parameter contexts were at the
time the flow version is being saved (minus sensitive values). On import of
this flow from registry to another NiFi instance, it will attempt to
create/update parameter contexts based on the snapshot in the flow being
imported, and any sensitive values would need to be populated after the
import.

On Thu, Nov 2, 2023 at 9:42 AM Mike Thomsen  wrote:

> I'm trying to track down the parameter sync issue I posted earlier.
>
> My understanding is that the parameters are stored in the Registry unless
> you configure your own parameter provider. Is that right? We're just using
> a Registry instance without any custom providers. Trying to figure out why
> we did not get the new parameters on the base context to sync.
>
> Thanks,
>
> Mike
>

Re: custom nar bundle fails to generate documentation - Asks for nifi dependencies

[Bryan Bende]

Hello,

Please try updating the version of the NAR Maven plugin you are using from
1.4.0 to the latest 1.5.1. I believe it will address this issue.

Your output shows:
org.apache.nifi:nifi-nar-maven-plugin:1.4.0:nar

Thanks,

Bryam

On Mon, Oct 30, 2023 at 2:14 PM Nguyen, Kyle 
wrote:

> Hi there.  I think my issue is related to this thread:
> https://www.mail-archive.com/dev@nifi.apache.org/msg17623.html
>
> And this SO:
> https://stackoverflow.com/questions/60670370/could-not-generate-extensions-documentation-when-creating-custom-controller-ser
>
>
>
> I have this in my nifi-*-nar pom.xml
>
>
>
> 
>
> 
>
>   org.apache.nifi
>
>   nifi-standard-services-api-nar
>
>   1.20.0
>
>   nar
>
> 
>
>
>
> Still it is failing and asks for dependencies not directly related to my
> project (I’m assuming they’re nifi deps).
>
>
>
> · javax.annotation:javax.annotation-api:jar:1.3.2
>
> · org.apache.taglibs:taglibs-standard-impl:jar:1.2.5
>
> · org.apache.taglibs:taglibs-standard-spec:jar:1.2.5
>
> · org.eclipse.jdt:ecj:jar:3.19.0
>
> · org.eclipse.jetty:apache-jsp:jar:9.4.50.v20221201
>
> · etc…
>
>
>
> [INFO] --- nifi-nar:1.4.0:nar (default-nar) @ nifi-bdcore-nar ---
>
> [INFO] Generating documentation for NiFi extensions in the NAR...
>
> [INFO] Found NAR dependency of
> org.apache.nifi:nifi-standard-services-api-nar:nar:1.20.0:compile
>
> [INFO] Found NAR dependency of
> org.apache.nifi:nifi-jetty-bundle:nar:1.20.0:compile
>
> [INFO] Found a dependency on version 1.20.0 of NiFi API
>
> [ERROR] Could not generate extensions' documentation
>
> org.apache.maven.plugin.MojoExecutionException: Failed to create Extension
> Documentation
>
> at org.apache.nifi.NarMojo.generateDocumentation (NarMojo.java:541)
>
> at org.apache.nifi.NarMojo.execute (NarMojo.java:512)
>
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
> (DefaultBuildPluginManager.java:126)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2
> (MojoExecutor.java:328)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute
> (MojoExecutor.java:316)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:212)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:174)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.access$000
> (MojoExecutor.java:75)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor$1.run
> (MojoExecutor.java:162)
>
> at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute
> (DefaultMojosExecutionStrategy.java:39)
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:159)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:105)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> (LifecycleModuleBuilder.java:73)
>
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
> (SingleThreadedBuilder.java:53)
>
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
> (LifecycleStarter.java:118)
>
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
>
> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
>
> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
>
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
>
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
>
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
>
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native
> Method)
>
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
>
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke (Method.java:566)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
> (Launcher.java:283)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
> (Launcher.java:226)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
> (Launcher.java:407)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.main
> (Launcher.java:348)
>
> Caused by: org.apache.maven.plugin.MojoExecutionException: Could not
> resolve local dependency
> org.eclipse.jetty:apache-jsp:jar:9.4.50.v20221201:compile
>
> at
> org.apache.nifi.extension.definition.extraction.ExtensionClassLoaderFactory.toURLs
> (ExtensionClassLoaderFactory.java:330)
>
> at
> org.apache.nifi.extension.definition.extraction.ExtensionClassLoaderFactory.createClassLoader
> (ExtensionClassLoaderFactory.java:272)
>
> at
> org.apache.nifi.extension.definition.extraction.ExtensionClassLoaderFactory.createClassLoader
> (ExtensionClassLoaderFactory.java:120)
>
> at
> 

Re: Unable to Start NiFi Cluster After Upgrading to Version 1.21.0 from 1.16.0

[Bryan Bende]

Hello,

We would need to see more information about the error besides that single
line, there should be a whole stacktrace that we would need to see.

Thanks,

Bryan

On Mon, Jul 31, 2023 at 9:49 AM Pallavi Metkar 
wrote:

> Hi Team,
>
> I have upgraded the Apache NiFi cluster from 1.16.0 to 1.21.0. After
> completing the upgrade NiFi is not working.
>
> I am getting below error for the same:
>
> o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during
> polling for external resources
>
> I had tried to resolve the error but was not able to do so.
> Can you please help?
>
> Thank you
>
> Regards
> Pallavi
>

Re: The question of "403 Forbidden"

[Bryan Bende]

Hello,

It looks like your user has READ permissions to the resource, but not WRITE
permissions. You will need to check your access policies related to the
processor and/or the parent process group hierarchy.

Thanks,

Bryan

On Mon, Jul 31, 2023 at 8:09 AM Zhiyi Ni  wrote:

> Hello, I have some questions and hope to get your help.
> According to the REST API, I performed the following operations in PostMan:
>
>
> 1. Request the following address to get access token
> |
> POST https://:/nifi-api/access/token
> with body:
> {
>   "username": xxx,
>   "password": xxx
> }
> |
>
> The request returned a cookie that allowed access to the REST API
>
>
> `__Secure-Authorization-Bearer=xxx;__Secure-Request-Token=`
>
>
>
>
> 2. When I visit the following address with this cookie to obtain basic
> information, I can get the expected response result
>
> |
> ①: GET https://:/nifi-api/process-groups/{group_id}
> ②: GET https://
> :/nifi-api/process-groups/{group_id}/processors
> |
>
>
>
>
> 3. But when I use this cookie to access the following address and try to
> create a new processor, there is a "403 Forbidden" exception
>
> |
> POST https://
> :/nifi-api/process-groups/{group_id}/processors
> with body:
> {
>   "permissions": {
> "canRead": true,
> "canWrite": true
>   },
>   "component": {
> "name": "GET_IC_COPPER_RESOURCES_MINING",
> "type": "org.apache.nifi.processors.mongodb.GetMongo",
> "bundle": {
>   "group": "org.apache.nifi",
>   "artifact": "nifi-mongodb-nar",
>   "version": "1.22.0"
> },
> "state": "STOPPED",
> "relationships": [
>   {
> "name": "failure",
> "autoTerminate": true,
> "retry": false
>   },
>   {
> "name": "original",
> "autoTerminate": true,
> "retry": false
>   },
>   {
> "name": "success",
> ,
> "autoTerminate": false,
> "retry": false
>   }
> ],
> "supportsParallelProcessing": true,
> "supportsEventDriven": false,
> "supportsBatching": false,
> "supportsSensitiveDynamicProperties": false,
> "persistsState": false,
> "restricted": false,
> "deprecated": false,
> "executionNodeRestricted": false,
> "multipleVersionsAvailable": false,
> "inputRequirement": "INPUT_ALLOWED",
> "config": {
>   "properties": {
> "mongo-client-service": null,
> "Mongo URI": "mongodb://hostxx:portxx",
> "Mongo Database Name": "xxx",
> "Mongo Collection Name": "",
> "ssl-context-service": null,
> "ssl-client-auth": "REQUIRED",
> "json-type": "Standard",
> "use-pretty-printing": "true",
> "mongo-charset": "UTF-8",
> "mongo-date-format": "-MM-dd HH:mm:ss",
> "get-mongo-send-empty": "false"
>   },
>   "schedulingPeriod": "5 sec",
>   "schedulingStrategy": "TIMER_DRIVEN",
>   "executionNode": "PRIMARY",
>   "penaltyDuration": "30 sec",
>   "yieldDuration": "1 sec",
>   "bulletinLevel": "WARN",
>   "runDurationMillis": 0,
>   "concurrentlySchedulableTaskCount": 1,
>   "lossTolerant": false,
>   "retryCount": 10,
>   "retriedRelationships": [
>
>   ],
>   "backoffMechanism": "PENALIZE_FLOWFILE",
>   "maxBackoffPeriod": "10 mins"
> },
> "validationErrors": [
>
> ],
> "validationStatus": "VALID",
> "extensionMissing": false
>   },
>   "inputRequirement": "INPUT_ALLOWED",
>   "operatePermissions": {
> "canRead": true,
> "canWrite": true
>   }
> }
> |
>
>
>
>
> And the response is:
>
> |
> 
>
>
> 
> 
> Error 403 Forbidden
> 
>
>
> 
> HTTP ERROR 403 Forbidden
> 
> 
> URI:
>
> /nifi-api/process-groups/80a631d9-d4cf-134b-a6b7-1ef07a3de334/processors
> 
> 
> STATUS:
> 403
> 
> 
> MESSAGE:
> Forbidden
> 
> 
> SERVLET:
> jerseySpring
> 
> 
>
>
> 
>
>
> 
> |
>
>
>
>
> How to solve the "HTTP ERROR 403 Forbidden" encountered in step 3?

Re: dynamic properties not recovering from ghost component state

[Bryan Bende]

Hi Nissim,

I'm wondering if we need to store the set of sensitiveDynamicPropertyNames
in flow.json or flow.xml. Currently when loading the flow we are recreating
this set by looking at the property values and seeing which properties are
dynamic and have an encrypted value. The problem you are highlighting is
that once the component is ghosted, the flow.json/flow.xml now has all
encrypted values and there is no way to get back to the original set of
sensitiveDynamicPropertyNames. For flow.json it would likely be in
VersionedConfigurableExtension where the other property fields are. For
flow.xml, it would require updates to the serialization/deserialization.
When serializing the sensitiveDynamicPropertyNames, it would have to always
retain the set as submitted when the processor was created/updated, and not
base it off the current component's descriptors, otherwise it will have the
same issue.

Curious to see what others think.

Thanks,

Bryan


On Fri, Jun 30, 2023 at 10:46 AM Nissim Shiman 
wrote:

> Hello apache nifi devs,
>
> I have run into an issue that appears to require a different way of
> handling ghost components than we are currently, and would like input
> before proceeding further.
>
>
> Specifically, this is related to
> https://issues.apache.org/jira/browse/NIFI-11570
>
>
> Besides the example in the ticket, a good case where this issue can be
> seen is with the ExecuteScript processor with two dynamic properties, one
> sensitive and one not.  On recovery from ghost component, both properties
> will be sensitive.
>
>
> The challenge is that on recovery from ghost component there no way of
> knowing if a dynamic property is sensitive or not, so it is left in the
> existing sensitive/encrypted state.
>
>
> There are solutions, but each is a shift from how nifi is currently
> working.
>
> 1.  Leave unencrypted properties in the clear when entering ghost
> component mode.  This would work, but we'd lose protections of why
> encryption was done in the first place, so probably a non-starter.
>
> 2.  When encrypting properties in ghost mode, have a custom encryption tag
> in flow.json.gz/flow.xml.gz.   i.e. instead of sensitive values starting
> with enc{   they would start with enc-ghost{   This will help when exiting
> ghost mode where we could now more easily determine why a value was
> encrypted in the first place, but is introducing new complexities in
> keeping track of why something is encrypted
>
>
> 3.  Leave handling of ghost components as is, but for a given component
> only allow Dynamic Properties to be non-sensitive or senstive, but not
> both.  This would affect ExecuteScript, InvokeHTTP (and others) which
> currently allow both and would be rolling back considerable work that was
> done to allow this, and we'd lose existing functionality, so also not
> likely.
>
>
> 4.  Leave as is.  It is not so common to have components in ghost state,
> and only some of them allow Dynamic Properties so this is an acceptable
> risk considering the alternatives.
>
> I am interested in community feedback and/or other suggestions to the
> current situation.
>
>
> Thanks,
>
>
> Nissim Shiman
>
>
>
>
>
>

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.1

[Bryan Bende]

+1 (binding)

Verified signatures and hashes
Built plugin on MacOs with Java 11.0.16 and Maven 3.9.2
Built multiple projects with 1.5.1 plugin and verified issues from
NIFI-11324 no longer occur.

Thanks!

On Thu, Jun 15, 2023 at 8:53 AM Peter Turcsanyi 
wrote:

> +1 (binding)
>
> Verified signatures and hashes.
> Built NiFi NAR Maven Plugin and then NiFi using the new plugin on macOS
> Ventura 13.2.1 and on Ubuntu 20.04 (Java: Adoptium Temurin 11.0.18+10,
> Maven: 3.9.2, empty local repo).
> Checked some NARs and ran NiFi. No issues found.
>
> Thanks for RMing Soma!
>
> Regards,
> Peter Turcsanyi
>
> On Thu, Jun 15, 2023 at 12:55 AM Nandor Soma Abonyi
>  wrote:
>
> > Hello Apache NiFi Community,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > NAR Maven Plugin 1.5.1.
> >
> > The source being voted upon, including signatures, digests, etc. can be
> > found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.5.1/
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
> >
> > The Git tag is nifi-nar-maven-plugin-1.5.1-rc1
> > The Git commit ID is 39fc959426ea405df6360969b55ae2adad47e1aa
> >
> >
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=39fc959426ea405df6360969b55ae2adad47e1aa
> >
> > Checksums of nifi-nar-maven-plugin-1.5.1-source-release.zip:
> > SHA256: 0ddc4efbfe504f9ed6477b8c572f2c6e5ba0c953e2e5b063bdfbd1f934eda6bf
> > SHA512:
> >
> a7281c8a3769db37e3491f3a5e54533b5f26bdcad99f8adb1e5609f1de17309aefb3d49eb9231e75a814e42566525b9afe4a11bda2c4ab48e8bab5a298b72b24
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/nsabonyi.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 3 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353009
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.5.1
> >
> > The vote will be open for 72 hours. Please download the release
> > candidate and evaluate the necessary items including checking hashes,
> > signatures, build from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because …
> >
> >
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Bryan Bende]

+1 Thanks Nandor, I agree it would be nice to get that fix out.

On Mon, Jun 12, 2023 at 10:05 AM Kevin Doran  wrote:

>  +1
>
> On Jun 12, 2023 at 05:35:57, Nandor Soma Abonyi
> 
> wrote:
>
> > Hello Apache NiFi Community,
> >
> > I'd like to initiate a discussion about the next release of
> > nifi-nar-maven-plugin.
> >
> > Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which
> was
> > fixed in [1]. Though it is the only change since the last release, I
> > consider it significant
> > enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.
> >
> > I would be happy to take on RM duties for this release.
> >
> > Do you agree it is time for a new release?
> >
> > Thanks,
> > Nandor Soma Abonyi
> >
> > [1] NIFI-11324  NAR
> > Plugin extension docs not correctly resolving overridden versions from
> > system properties
> >
> >
>

Re: DeleteHDFS behavior when idle

[Bryan Bende]

The processor has @TriggerWhenEmpty so it is going to keep executing
regardless of whether the incoming queue has data or not. I believe this
was done early on for some processors that used Kerberos in order to allow
the processor to have a chance to renew the Kerberos ticket, however we
since moved away from need to do this, so unless there is another reason
for having that, I would think it can be removed.

On Mon, Jun 12, 2023 at 9:25 AM Mark Payne  wrote:

> Isha,
>
> If you have an incoming connection, and you’re seeing this, then it’s a
> bug. If there is no incoming connection and this processor is used as a
> source processor, it’s normal. Either way, it has rather little overhead,
> and you can further reduce the overhead by increasing the Yield Duration in
> settings. This is how long it will wait between invocations if there’s
> nothing for it to do.
>
> Either way, best to file a Jira, though, to address the behavior for
> running unnecessarily when there’s an incoming Connection.
>
> Thanks
> -Mark
>
>
> > On Jun 12, 2023, at 8:36 AM, Isha Lamboo 
> wrote:
> >
> > Hi all,
> >
> > I have a question about behavior I see on one of our NiFi 1.18 clusters
> that has a lot of xHDFS processors. When I look at the number of tasks in
> the summary, the DeleteHDFS processors have a very high number (800-1000+)
> of tasks even if they have nothing in their incoming queues. The PutHDFS
> and FetchHDFS in contrast have no tasks listed when they have no files in
> the incoming queues. Even though the tasks take very little time (less than
> 100 millis per 5 mins), I’m wondering whether this causes problems when the
> cluster is heavily loaded during peak hours.
> >
> > Is this a bug or some feature related to deleting files? Should I submit
> a ticket?
> >
> > Thanks,
> >
> > Isha
>
>

Re: "Broken pipe" when trying to use the CLI

[Bryan Bende]

Hello,

You can add -verbose to the command to see a full error trace, however I'm
not sure it will tell you much more than the broken pipe message.

Broken pipe generally means some kind of network issue, typically it means
the other side of the connection closed the connection while the current
side was trying to read or write.

Ultimately I don't think it is really a CLI issue.

-Bryan


On Tue, May 2, 2023 at 3:51 PM Nguyen, Kyle 
wrote:

> Hi.
>
>
>
> We are getting “Broken pipe” when trying nifi-toolkit.  How would we go
> about debugging this?  Any help is appreciated!  Thanks!
>
>
>
> root@bizdev-nifi-dev:/opt/nifi/nifi-toolkit-current/bin# ./cli.sh
>
> Session loaded from /root/.nifi-cli.config
>
> #> session show
>
> Current Session:
>
> nifi.props = /opt/nifi/nifi-current/conf/dev.nifi-cli.nifi.properties
>
> #> nifi pg-list
>
> ERROR: Error executing command 'pg-list' : Broken pipe (Write failed)
>
>
>
> Per documentation for cli toolkit:
> https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#security-configuration
>
>
>
> Currently the CLI supports authenticating with a client
> certificate and an optional proxied-entity. A common scenario would be
> running the CLI from one of the nodes where NiFi or NiFi Registry is
> installed, which allows the CLI to use the same keystore and truststore as
> the NiFi/NiFi Registry instance.
>
>
>
> We have a keystore and truststore to secure our instance:
>
>
>
> root@bizdev-nifi-dev:/opt/nifi/nifi-current/conf# cat nifi.properties |
> grep -i --color=auto jks
>
> nifi.security.keystore=/opt/certs/keystore.jks
>
> nifi.security.keystoreType=JKS
>
> nifi.security.truststore=/opt/certs/truststore.jks
>
> nifi.security.truststoreType=JKS
>
> root@bizdev-nifi-dev:/opt/nifi/nifi-current/conf# grep -i --color=auto
> jks login-identity-providers.xml
>
> /opt/certs/keystore.jks
>
> JKS
>
> /opt/certs/truststore.jks
>
> JKS
>
>
>
> Then inside my cli-toolkit properties file, I have:
>
>
>
> root@bizdev-nifi-dev:/opt/nifi/nifi-current/conf# cat
> dev.nifi-cli.nifi.properties
>
> baseUrl=https://bizdev-nifi-dev.nonprd.aws.mlp.com
>
> keystore=/opt/certs/keystore.jks
>
> keystoreType=JKS
>
> keystorePasswd=*
>
> keyPasswd=*
>
> truststore=/opt/certs/truststore.jks
>
> truststoreType=JKS
>
> truststorePasswd=
>
>
>
> [image: cid:17261f6184a4cff311]
>
> *Kyle Nguyen*
> Corporate Technology, Software Engineer
>
>
>
> *Millennium Management LLC*
>
> 399 Park Avenue  |  New York, NY 10022
>
>  +1.212.708.1366  |  +1.929.837.1788
>
> mlp.com 
>
>
>
>
>
> ##
>
> The information contained in this communication is confidential and
>
> may contain information that is privileged or exempt from disclosure
>
> under applicable law. If you are not a named addressee, please notify
>
> the sender immediately and delete this email from your system.
>
> If you have received this communication, and are not a named
>
> recipient, you are hereby notified that any dissemination,
>
> distribution or copying of this communication is strictly prohibited.
> ##
>
>

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Bryan Bende]

You are right, I didn't actually try to download it before, that is strange.

On Mon, Apr 10, 2023 at 1:07 PM Mike Thomsen  wrote:
>
> When I clicked on it, I got a 404. Weird.
>
> On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende  wrote:
>
> > The Azure NAR looks like it is there:
> >
> > https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/
> >
> > On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen 
> > wrote:
> > >
> > > It’s failing on the azure one which is why I am raising the issue. I
> > know we are still pushing that because it’s part of the convenient binary.
> > >
> > > Sent from my iPhone
> > >
> > > > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> > > >
> > > > Hello
> > > >
> > > > Pretty sure all the normal stuff/paths followed and uploaded.  It
> > might be
> > > > that we're not pushing Atlas nars anymore?
> > > >
> > > > I dont recall but that is possible.
> > > >
> > > > Thanks
> > > >
> > > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen 
> > wrote:
> > > >>
> > > >> Getting this error on two different machines on two different
> > networks:
> > > >>
> > > >> ownloaded from central:
> > > >>
> > > >>
> > https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> > > >> (26 MB at 183 kB/s)
> > > >>
> > > >> Downloaded from central:
> > > >>
> > > >>
> > https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> > > >> (49 MB at 332 kB/s)
> > > >>
> > > >> Downloaded from central:
> > > >>
> > > >>
> > https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> > > >> (74 MB at 491 kB/s)
> > > >>
> > > >> [*INFO*]
> > > >>
> > **
> > > >>
> > > >> [*INFO*] *BUILD FAILURE*
> > > >>
> > > >> [*INFO*]
> > > >>
> > **
> > > >>
> > > >> [*INFO*] Total time:  02:54 min
> > > >>
> > > >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
> > > >>
> > > >> [*INFO*]
> > > >>
> > **
> > > >>
> > > >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
> > > >> resolve dependencies for project
> > org.apache.nifi:nifi-assembly:pom:1.21.0:
> > > >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
> > > >> central (https://repo.maven.apache.org/maven2
> > > >> <https://repo.maven.apache.org/maven2>)* -> *[Help 1]*
> > > >>
> > > >> [*ERROR*]
> > > >>
> > > >> [*ERROR*] To see the full stack trace of the errors, re-run Maven
> > with the
> > > >> *-e* switch.
> > > >>
> > > >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug
> > logging.
> > > >>
> > > >> [*ERROR*]
> > > >>
> > > >> [*ERROR*] For more information about the errors and possible
> > solutions,
> > > >> please read the following articles:
> > > >>
> > > >> [*ERROR*] *[Help 1]*
> > > >>
> > > >>
> > http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> > > >>
> >

Re: Are all 1.21 NARs uploaded as Maven artifacts?

[Bryan Bende]

The Azure NAR looks like it is there:

https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/

On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen  wrote:
>
> It’s failing on the azure one which is why I am raising the issue. I know we 
> are still pushing that because it’s part of the convenient binary.
>
> Sent from my iPhone
>
> > On Apr 10, 2023, at 12:26 PM, Joe Witt  wrote:
> >
> > Hello
> >
> > Pretty sure all the normal stuff/paths followed and uploaded.  It might be
> > that we're not pushing Atlas nars anymore?
> >
> > I dont recall but that is possible.
> >
> > Thanks
> >
> >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen  
> >> wrote:
> >>
> >> Getting this error on two different machines on two different networks:
> >>
> >> ownloaded from central:
> >>
> >> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar
> >> (26 MB at 183 kB/s)
> >>
> >> Downloaded from central:
> >>
> >> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-other-graph-services-nar/1.21.0/nifi-other-graph-services-nar-1.21.0.nar
> >> (49 MB at 332 kB/s)
> >>
> >> Downloaded from central:
> >>
> >> https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-atlas-nar/1.21.0/nifi-atlas-nar-1.21.0.nar
> >> (74 MB at 491 kB/s)
> >>
> >> [*INFO*]
> >> **
> >>
> >> [*INFO*] *BUILD FAILURE*
> >>
> >> [*INFO*]
> >> **
> >>
> >> [*INFO*] Total time:  02:54 min
> >>
> >> [*INFO*] Finished at: 2023-04-10T12:20:08-04:00
> >>
> >> [*INFO*]
> >> **
> >>
> >> [*ERROR*] Failed to execute goal on project nifi-assembly: *Could not
> >> resolve dependencies for project org.apache.nifi:nifi-assembly:pom:1.21.0:
> >> Could not find artifact org.apache.nifi:nifi-azure-nar:nar:1.21.0 in
> >> central (https://repo.maven.apache.org/maven2
> >> )* -> *[Help 1]*
> >>
> >> [*ERROR*]
> >>
> >> [*ERROR*] To see the full stack trace of the errors, re-run Maven with the
> >> *-e* switch.
> >>
> >> [*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging.
> >>
> >> [*ERROR*]
> >>
> >> [*ERROR*] For more information about the errors and possible solutions,
> >> please read the following articles:
> >>
> >> [*ERROR*] *[Help 1]*
> >>
> >> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> >>

Re: Usage Documentation for Custom Processors

[Bryan Bende]

Hello,

Documentation for a component is generated automatically from the component
itself by using available API methods and the annotations on the class [1].
For example, documentation on a processor will use the
getPropertyDescriptors and getRelationship methods of the component, as
well as annotations like @CapabilityDescription and @Tags, etc.

Thanks,

Bryan

[1]
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#documenting-a-component


On Tue, Apr 4, 2023 at 10:54 AM Matthew Baine 
wrote:

> Hi Bryan,
>
> Sorry, on a separate note, what would be the best way to set up Usage
> Documentation for a custom processor?
>
> [image: image.png]
>
> We can't seem to get this right with the information online and on the
> Nifi developer guide (
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html). Our
> custom processors seem to only publish documentation of the native
> processors.
>
>
> Kind Regards,
> Matthew
>
> On Tue, 4 Apr 2023 at 13:54, Matthew Baine 
> wrote:
>
>> Hi Bryan,
>>
>> Sorry for the delayed response, and thank you so much for the feedback!
>>
>> We will attempt the advised approach and revert if we run into any
>> trouble.
>>
>> Thanks again!
>>
>> Regards,
>>
>> On Thu, 30 Mar 2023 at 16:49, Bryan Bende  wrote:
>>
>>> Hello,
>>>
>>> This might not give you exactly what you want, but the Minifi Toolkit
>>> already has the ability to transform the JSON snapshot from registry,
>>> there are actually two commands:
>>>
>>> "transform" - for XML templates
>>> "transform-vfs" - for versioned flow snapshot (JSON from registry) [1]
>>>
>>> It doesn't pull the snapshot from registry directly, so you would have
>>> to script something to download the snapshot and then run
>>> transform-vfs.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> [1]
>>> https://github.com/apache/nifi/blob/main/minifi/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMain.java#L62
>>>
>>> On Thu, Mar 30, 2023 at 10:22 AM Simeon Wentzel 
>>> wrote:
>>> >
>>> > Dear Nifi dev team
>>> >
>>> > Can you add extended functionality to the MiNiFi toolkit to extract a
>>> flow
>>> > from the NiFi Registry software and convert it to the appropriate
>>> conf.yml
>>> > file?
>>> >
>>> > We have found a limitation regarding the conversion in the minifi
>>> toolkit
>>> > that it can only convert the .xml file template extracted from a Nifi
>>> > canvas on Java version 8, it can not do the conversion on java 11 that
>>> we
>>> > have migrated to.
>>> >
>>> > Although extracting the flow as a template out of nifi and then
>>> converting
>>> > it to the conf.yaml file works we find it a bit cumbersome because we
>>> can
>>> > not implement it in our pipeline to automate the process.
>>> >
>>> > By allowing the minifi toolkit to pull a flow from the Nifi registry
>>> and
>>> > then convert it will give us the functionality to add this in our
>>> Jenkins
>>> > pipeline to build individual docker containers for each of our flows.
>>> >
>>> > Regards
>>> > Simeon
>>> > DevOps Engineer
>>>
>>
>>
>> --
>>
>>
>> *Matthew Baine | *DevOps Engineer
>>
>> *Johannesburg Head Office*
>>
>> E: matt...@airvantage.co.za | M: +27 (0) 71053 9012 <+27710539012>
>>
>> T: +27 (0) 11 100 1880 <+270111001880> | W: www.airvantage.co.za
>>
>> *Skype: matthew.baine57*
>>
>
>
> --
>
>
> *Matthew Baine | *DevOps Engineer
>
> *Johannesburg Head Office*
>
> E: matt...@airvantage.co.za | M: +27 (0) 71053 9012 <+27710539012>
>
> T: +27 (0) 11 100 1880 <+270111001880> | W: www.airvantage.co.za
>
> *Skype: matthew.baine57*
>

Re: Add functionality to MiNiFi toolkit

[Bryan Bende]

Hello,

This might not give you exactly what you want, but the Minifi Toolkit
already has the ability to transform the JSON snapshot from registry,
there are actually two commands:

"transform" - for XML templates
"transform-vfs" - for versioned flow snapshot (JSON from registry) [1]

It doesn't pull the snapshot from registry directly, so you would have
to script something to download the snapshot and then run
transform-vfs.

Thanks,

Bryan

[1] 
https://github.com/apache/nifi/blob/main/minifi/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMain.java#L62

On Thu, Mar 30, 2023 at 10:22 AM Simeon Wentzel  wrote:
>
> Dear Nifi dev team
>
> Can you add extended functionality to the MiNiFi toolkit to extract a flow
> from the NiFi Registry software and convert it to the appropriate conf.yml
> file?
>
> We have found a limitation regarding the conversion in the minifi toolkit
> that it can only convert the .xml file template extracted from a Nifi
> canvas on Java version 8, it can not do the conversion on java 11 that we
> have migrated to.
>
> Although extracting the flow as a template out of nifi and then converting
> it to the conf.yaml file works we find it a bit cumbersome because we can
> not implement it in our pipeline to automate the process.
>
> By allowing the minifi toolkit to pull a flow from the Nifi registry and
> then convert it will give us the functionality to add this in our Jenkins
> pipeline to build individual docker containers for each of our flows.
>
> Regards
> Simeon
> DevOps Engineer

Re: [discuss] NiFi support for Hadoop ecosystem components

[Bryan Bende]

I lean towards option 2 with the caveat that maybe we don't have to
retain every Hadoop related component when creating this separate set
of components. Mainly I'm thinking that Hive has been the most
problematic to maintain so maybe that is dropped all together. I think
it would be unfortunate to not have publicly available HDFS
processors.

On Fri, Mar 24, 2023 at 3:23 PM Matt Burgess  wrote:
>
> As one of the small number of people that fight the battle, I like the
> idea of Option 1 (full disclosure: I work for a vendor). From a
> community standpoint (I'm on the PMC) I'm not strongly opposed to
> Option 2 although I wouldn't want to be the one managing and releasing
> the artifacts :) Having said that, unless it remained maintained and
> released, I feel like it would just be a component graveyard (or maybe
> more like the Apache Attic), in which case it seems unnecessary and
> that's why I'm behind Option 1. Interested to hear others' thoughts of
> course.
>
> Thanks,
> Matt
>
> On Fri, Mar 24, 2023 at 2:07 PM Joe Witt  wrote:
> >
> > Team,
> >
> > For the full time NiFi has been in Apache we've built with support for
> > various Hadoop ecosystem components like HDFS, Hive, HBase, others,
> > and more recently formats/serialization modes like necessary for
> > Parquet, Orc, Iceberg, etc..
> >
> > All of these things however present endless challenges with
> > compatibility across different versions (Hive being the most difficult
> > by far), vendors (hadoop vendors, cloud vendors, etc..).  And also
> > super notably the incredible number of dependencies, dependency
> > conflicts, inclusions/exclusions, old log libs, vulnerability updates,
> > etc..  And last but certainly not least a big reason why our build has
> > grown so much.
> >
> > We have a couple options:
> > 1. Deprecate these components in NiFi 1.x and drop them entirely in
> > NiFi 2.x.  Leave this as a problem for vendors to deal with.  NiFi
> > users interacting with such components are nearly exclusively doing so
> > with vendors anyway.
> >
> > 2. Remove the components from NiFi main code line and create a
> > separate repo for 'nifi-hadoop-extensions'.  We manage those
> > independently and release them periodically.  They would be available
> > for people to grab the nars if they want to use them.  We include none
> > of them in the convenience binary going forward by default.
> >
> > 3. Change nothing.  Continue to battle with the above listed items.
> > This is admittedly a bit of a non-option.  We can't keep spending the
> > same time/energy on these we have.  It is a very small number of
> > people that fight this battle.
> >
> > Look forward to hearing thoughts on these options or others we might 
> > consider.
> >
> > Thanks

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.4.0

[Bryan Bende]

+1 binding

Everything checks out, thanks Kevin!

On Wed, Feb 1, 2023 at 5:52 AM Ferenc Erdei  wrote:
>
> +1 Non binding
>
>  Verified signatures and hashes
> - Built on OSX 13.1, with OpenJDK Runtime Environment (Temurin)(build 
> 1.8.0_362-b09) Maven 3.8.3
> - Built NiFi with the new plugin version
> - Started nifi and minifi and ran some simple flow
>
> > On 2023. Jan 31., at 1:12, Joe Witt  wrote:
> >
> > +1 binding
> >
> > Full clean build with contrib check
> > Signature
> > Hash
> > License/Notice
> > Intended changes in place
> > Full clean build with contrib check for nifi including all optional profiles
> > Full clean build with contrib check excluding optional profiles (size still
> > within 1.5G hard limit)
> >
> > Thanks!
> >
> > On Mon, Jan 30, 2023 at 4:19 PM Kevin Doran  wrote:
> >
> >> Hello Apache NiFi Community,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> >> NiFi NAR Maven Plugin 1.4.0.
> >>
> >> The source being voted upon, including signatures, digests, etc. can
> >> be found at:
> >>
> >> https://repository.apache.org/content/repositories/orgapachenifi-1219/org/apache/nifi/nifi-nar-maven-plugin/1.4.0/
> >>
> >> The Git tag is nifi-nar-maven-plugin-1.4.0-RC1
> >> The Git commit ID is 09d7bb9ff679d0eed9feaa066d2cbdd347a20204
> >>
> >> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=09d7bb9ff679d0eed9feaa066d2cbdd347a20204
> >>
> >> Checksum of nifi-nar-maven-plugin-1.4.0-source-release.zip
> >>
> >> SHA512:
> >> d48bfde8a5bab8a17b320889297c09efa162a7017db2268e274db626926ff9b58173abab54db4e1a50b5664fc86a9501ce2ef267b3e3b768eb80ef5c929860c9
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/kdoran.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 6 issues were closed/resolved for this release:
> >>
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352124
> >>
> >> Release note highlights can be found here:
> >>
> >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.4.0
> >>
> >> The vote will be open for 72 hours. Please download the release
> >> candidate and evaluate the necessary items including checking hashes,
> >> signatures, build from source, and test. Then please vote:
> >>
> >> [ ] +1 Release this package as nifi-nar-maven-plugin-1.4.0
> >> [ ] +0 no opinion
> >> [ ] -1 Do not release this package because ...
> >>
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.4.0

[Bryan Bende]

Thanks Kevin.

Just to clarify, the Maven issues mentioned here don't sound specific
to the NAR plugin, but sound like general Maven issues? or am I
mis-interpreting?

On Wed, Jan 25, 2023 at 6:56 PM Joe Witt  wrote:
>
> Thanks Kevin.  Sounds good.
>
> If you can start that soon I can draft 1.20 off that..
>
> thanks
>
> On Wed, Jan 25, 2023 at 4:46 PM Kevin Doran  wrote:
>
> >  Hi Harry,
> >
> > Thanks for the heads up. If you can share those findings, there's a few of
> > us on this list that could help look into them and see if NiFi can do
> > anything to work around potential Maven issues.
> >
> > Cheers,
> > Kevin
> >
> > On Jan 25, 2023 at 18:42:07, Harry Clarke 
> > wrote:
> >
> > > I'm not sure how to view what's been worked on since the last release,
> > but
> > > my team did notice an issue with 3.8.1 of Maven that isn't present in
> > 3.6.1.
> > >
> > > It's hard to nail down exactly what's going on, but it looked to be that
> > > plugin repository proxy settings weren't being honoured by Maven within
> > the
> > > settings.xml files, and pom.xml repo declarations were taking precedence.
> > > Of course, this fails immediately if you're behind a firewall.
> > >
> > > I'll try to dig up where we got to with our investigation, but this was a
> > > blocker for us moving to Java 17.
> > > 
> > > From: Kevin Doran 
> > > Sent: 25 January 2023 23:34
> > > To: dev@nifi.apache.org 
> > > Subject: Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.4.0
> > >
> > > Hi all,
> > >
> > > The issues raised on this thread have all been addressed, so I’d like to
> > > revive this goal to release NiFi NAR Maven Plugin 1.4.0 soon.
> > >
> > > Unless anyone is aware of additional items to include, I plan to start
> > > preparing a release candidate this week.
> > >
> > > Thanks,
> > > Kevin
> > >
> > > On Nov 30, 2022 at 14:28:00, Kevin Doran  wrote:
> > >
> > > Thanks, Bryan! There is no urgency around this release, so I'm happy to
> > >
> > > wait for that.
> > >
> > >
> > > On Nov 30, 2022 at 14:24:02, Bryan Bende  wrote:
> > >
> > >
> > > > Thanks Kevin!
> > >
> > > >
> > >
> > > > There is actually one change I was planning to start working on, and
> > >
> > > > since we don't release the NAR plugin very frequently, I would like to
> > >
> > > > try and get it in before this release.
> > >
> > > >
> > >
> > > > I created this JIRA [1] for the issue, and I can report back here once
> > >
> > > > I start working on it to see if it looks like it will still be
> > >
> > > > something to wait on.
> > >
> > > >
> > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-10915
> > >
> > > >
> > >
> > > >
> > >
> > > > On Wed, Nov 30, 2022 at 9:43 AM David Handermann
> > >
> > > >  wrote:
> > >
> > > >
> > >
> > > >
> > >
> > > > Mark,
> > >
> > > >
> > >
> > > >
> > >
> > > > The dependency duplication detection is a new optional goal of the NAR
> > >
> > > >
> > >
> > > > plugin. The basic purpose is to detect unnecessary dependencies in the
> > >
> > > >
> > >
> > > > compile scope, which are already provided from a parent NAR dependency.
> > >
> > > >
> > >
> > > >
> > >
> > > > For example, the nifi-standard-service-api-nar includes the
> > >
> > > >
> > >
> > > > nifi-ssl-context-service-api library. The
> > > nifi-web-client-provider-service
> > >
> > > >
> > >
> > > > depends on nifi-ssl-context-service-api, and identifies it correctly
> > with
> > >
> > > >
> > >
> > > > the provided scope in the Maven configuration. The
> > >
> > > >
> > >
> > > > nifi-web-client-provider-service-nar bundles
> > >
> > > >
> > >
> > > > nifi-web-client-provider-service, and depends on
> > >
> > > >
> > >
> > > > nifi-s

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Bryan Bende]

The plan as I understand it is not to diverge and create separate feature
development on the 1.x line, so I would expect all PRs to continue to be
submitted only to main. We would release 1.x as needed with major bug fixes
or critical security updates, and these would be cherry-picked and/or
backported as necessary, mostly without the need for PRs, the same as we
would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a
maintenance line like (1.19.x). For precedent, we followed this same
approach going from the 0.x line to 1.0.0 and there wasn't any major issue.

On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler  wrote:

>  It was also mentioned in another thread that we need to have agreement on
> our explicit strategy and support for 1.x going forward, did that happen?
>
> From: Otto Fowler  
> Reply: Otto Fowler  
> Date: January 10, 2023 at 07:02:34
> To: dev@nifi.apache.org  
> Subject:  Re: [discuss] NiFi 1.20 and NiFi 2.0
>
> There needs to be an update to the contributing guide as to how to submit
> PRs to 1.x or 2.x etc.
>
> From: Joe Witt  
> Reply: dev@nifi.apache.org  
> Date: January 9, 2023 at 15:53:16
> To: dev@nifi.apache.org  
> Subject:  [discuss] NiFi 1.20 and NiFi 2.0
>
> Team,
>
> As David mentioned in [1] following a successful NiFi 2.0 release goal
> planning - we are now going to start moving the 'main' line to be the NiFi
> 2.0 line which will allow for the key work to take place. We will also
> move niFi 1.x to its appropriate support line.
>
> It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have
> work in there including security items so it is time to make a release.
> The intent then is to initiate 1.20 and immediate after that change 'main'
> to 2.0.
>
> Going forward then all work on the 1.x line should be focused on
> maintaining existing features, dependencies, and helping 1.x users migrate
> to the 2.x line. Otherwise, new feature work will happen on 'main' as it
> normally does and will come out in the NiFi 2.x release line.
>
> Please flag key outstanding items as we narrow down the release candidate
> for NiFi 1.20.
>
> Thanks
> Joe
>
> [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz
>

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[Bryan Bende]

Hello,

The purpose of displayName is to allow changing the name shown in the
UI without breaking existing flows, meaning it still uses the original
name behind the scenes in flow xml/json, but just shows a different
name in the UI, so the primary purpose is for them to be different.

Since it was hard to know what the real name was, a recent release
added a new column in the documentation called "API Name". Example for
ConsumeKafka_2_6 processor:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-2-6-nar/1.19.1/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_2_6/index.html

Thanks,

Bryan

On Mon, Dec 12, 2022 at 1:21 PM Mathew Kiprop  wrote:
>
> Hello all,
>
> I hope this is the right thread to raise this.
> I have recently been working with nipyapi[1] creating flows and found
> challenge when configuring components(processors/CS) properties whose
> PropertyDescriptor$displayName is different than PropertyDescriptor$name.
> I had to use browser inspection tools to see how the request is submitted
> by NiFi fronted,  alternatively checking the name in the source code.
>
> Moving to 2.0, would it be good to standardize on PropertyDescriptor$name
> and displayname?
>
> Thanks,
> Mathew.
> [1] https://github.com/Chaffelson/nipyapi
>
> On Mon, 12 Dec 2022 at 19:46, David Handermann 
> wrote:
>
> > Hi Isha,
> >
> > As the scope of the proposed release goals does not include changing the
> > Site-to-Site protocol, that setup should work. Any kind of breaking
> > compatibility changes will need to be documented, but the general goal is
> > to make the upgrade as seamless as possible to encourage adoption.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Dec 12, 2022 at 4:12 AM Isha Lamboo <
> > isha.lam...@virtualsciences.nl>
> > wrote:
> >
> > > Hi all,
> > >
> > > This may be too basic/self-explanatory to count as a goal at all, but
> > will
> > > site-to-site interoperability between NiFi 1.x and NiFi 2.x nodes will be
> > > preserved?
> > >
> > > Regards,
> > >
> > > Isha
> > >
> > > -Oorspronkelijk bericht-
> > > Van: David Handermann 
> > > Verzonden: zondag 11 december 2022 04:08
> > > Aan: Otto Fowler 
> > > CC: dev@nifi.apache.org
> > > Onderwerp: Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0
> > >
> > > Hi Otto,
> > >
> > > Thanks for the reply, that is a good question.
> > >
> > > There is certainly a need to maintain the current version 1 branch for
> > > some amount of time, but the exact amount of time will need to be
> > > determined.
> > >
> > > Point 10 of the Proposed Release Goals includes implementing migration
> > > tools, which will have to be implemented in subsequent version 1
> > releases,
> > > so support for version 1 would not go away any time soon. It will become
> > > more difficult to maintain libraries as time goes on, but we should also
> > > identify some strategies for subsequent maintenance releases.
> > >
> > > I anticipate the need for future votes to sunset version 1, but that
> > > should not occur until there has been significant work on version 2 and
> > > associated migration tools, with documentation.
> > >
> > > The main purpose of the 2.0 Proposed Release Goals is to focus that
> > scope,
> > > and as you noted, we should give some parallel consideration to scoping
> > for
> > > version 1 releases.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sat, Dec 10, 2022 at 8:42 PM Otto Fowler 
> > > wrote:
> > >
> > > > Sorry to be late to this, the goals seem great. The question that
> > > > comes to my mind is will the current 1.x line will be maintained?
> > > >
> > > > That may be a parallel issue to the goals, but it is important if we
> > > > are dropping support for Java versions.
> > > >
> > > > I would think that *some* position on that has to be decided and
> > > > communicated ( if not voted on ).
> > > >
> > > >
> > >
> >

Re: [VOTE] Adopt NiFi 2.0 Proposed Release Goals

[Bryan Bende]

+1 (binding)

On Mon, Dec 12, 2022 at 12:27 PM Chris Sampson
 wrote:
>
> +1 (non-binding)
>
>
> Cheers,
>
> ---
> Chris Sampson
> IT Consultant
> chris.samp...@naimuri.com
>
>
> > On 12 Dec 2022, at 17:02, David Handermann  
> > wrote:
> >
> > Team,
> >
> > Following positive feedback on NiFi 2.0 Proposed Release Goals [1] on the
> > recent discussion thread [2], I am calling this vote to adopt the following
> > as Release Goals for NiFi 2.0:
> >
> > 1. Remove Java 8 support and require Java 11
> > 2. Remove deprecated components
> > 3. Remove deprecated component properties
> > 4. Remove components integrating with unmaintained services
> > 5. Remove compatibility classes and methods
> > 6. Remove flow.xml.gz in favor of flow.json.gz
> > 7. Remove duplicative features
> > 8. Upgrade internal Java API references
> > 9. Reorganize standard components
> > 10. Implement migration tools for upgrading flows
> >
> > A positive vote indicates agreement on these goals and the initiation of
> > the following actions:
> >
> > 1. Rename NiFi 2.0 Proposed Release Goals to NiFi 2.0 Release Goals
> > 2. Create version 1 branch in Git for subsequent support releases on the
> > version 1 series
> > 3. Update the current main branch in Git to version 2.0.0-SNAPSHOT
> >
> > The vote will be open for 72 hours and follow standard procedures for
> > release votes.
> >
> > Please review the linked goals and discussions for background.
> >
> > [ ] +1 Adopt NiFi 2.0 Release Goals
> > [ ] +0 No opinion
> > [ ] -1 Do not adopt NiFi 2.0 Release Goals for the following reasons...
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals
> > [2] https://lists.apache.org/thread/xo77p9t3xg4k70356xrqbdg4m9sg7sf8
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.4.0

[Bryan Bende]

Thanks Kevin!

There is actually one change I was planning to start working on, and
since we don't release the NAR plugin very frequently, I would like to
try and get it in before this release.

I created this JIRA [1] for the issue, and I can report back here once
I start working on it to see if it looks like it will still be
something to wait on.

[1] https://issues.apache.org/jira/browse/NIFI-10915


On Wed, Nov 30, 2022 at 9:43 AM David Handermann
 wrote:
>
> Mark,
>
> The dependency duplication detection is a new optional goal of the NAR
> plugin. The basic purpose is to detect unnecessary dependencies in the
> compile scope, which are already provided from a parent NAR dependency.
>
> For example, the nifi-standard-service-api-nar includes the
> nifi-ssl-context-service-api library. The nifi-web-client-provider-service
> depends on nifi-ssl-context-service-api, and identifies it correctly with
> the provided scope in the Maven configuration. The
> nifi-web-client-provider-service-nar bundles
> nifi-web-client-provider-service, and depends on
> nifi-standard-service-api-nar. If the nifi-ssl-context-service-api was not
> marked as provided, the new duplication detection goal would flag the
> unnecessary inclusion of the nifi-ssl-context-service-api.
>
> The duplication detection will help avoid including unnecessary
> dependencies, and also avoid unexpected runtime behavior. The NiFi NAR
> class loading hierarchy uses libraries from the parent NAR at runtime, so
> avoiding unnecessary dependency inclusion is important for these reasons.
> The goal is optional, and will require additional changes to enable by
> default in NiFi builds, but it should be very helpful for future releases.
>
> Regards,
> David Handermann
>
> On Wed, Nov 30, 2022 at 7:36 AM Mark Bean  wrote:
>
> > Sounds great Kevin. Thanks!
> >
> > Can you give a little more detail on the dependency duplication detection?
> > How does it work? Does it detect different versions of the same dependency?
> > Is it detecting duplicates only within a given NAR or across multiple NARs?
> >
> > Thanks,
> > Mark
> >
> >
> > On Tue, Nov 29, 2022 at 4:18 PM Kevin Doran  wrote:
> >
> > > Hi all,
> > >
> > > There’s been a few improvements and bug fixes to the NAR Maven Plugin.
> > One
> > > nice new feature is a new maven goal that detects duplicate dependencies
> > in
> > > NARs. Another contribution improves our NiFi build reproducibility.
> > >
> > > Given all this, I’d like to release a new version of the plugin that we
> > can
> > > start using in NiFi. As this includes a feature, this will be a minor
> > > version bump (1.4.0).
> > >
> > > I’m happy to RM. There are two outstanding PRs, and if there are no
> > > objections on this thread, I’ll wait for those to be merged and then
> > > prepare a release candidate.
> > >
> > > Thanks,
> > > Kevin
> > >
> >

Re: Git questions for supplying a contribution

[Bryan Bende]

You would need to do the "Clone from Fork" step, having first forked
the repo in Github.

Then your "origin" would be your fork, and you can push anything you
want to your fork.

You won't have permissions to push anything to the apache repo with
your current setup.

On Thu, Sep 29, 2022 at 11:54 AM Joe Witt  wrote:
>
> Dan
>
> You can think of 'upstream' as your 'origin'.
>
> You want to add your mirror in there.  You'd submit your code change
> in a branch to your fork/mirror.  And from that you'd request/submit a
> pull request to NiFi.  This is important because you wont be able to
> commit to the apache/nifi codebase for now (but if you become a
> committer you can - still though we all use the model I mention).
>
> Thanks
>
> On Thu, Sep 29, 2022 at 8:36 AM Dan S  wrote:
> >
> > Based on ContributorGuide-Cloneacopyoftherepository
> > I
> > had initially run
> > git clone https://github.com/apache/nifi.git and then I believe in error I
> > also ran
> > git remote add upstream https://github.com/apache/nifi.git
> >
> > Now when I run git remote -v
> > I have
> > origin https://github.com/apache/nifi.git (fetch)
> > origin https://github.com/apache/nifi.git (push)
> > upstream https://github.com/apache/nifi.git (fetch)
> > upstream https://github.com/apache/nifi.git (push)
> >
> > The instructions on ContributorGuide-Supplyingacontribution
> > 
> > for
> > a PR seem to be when the clone was from a fork (git clone
> > g...@github.com: > name>/nifi.git) and from a mirror.
> >
> > For my setup can I still run
> >
> > git push origin  ?

Re: [VOTE] Release Apache NiFi 1.17.0 (RC2)

[Bryan Bende]

+1 (binding)

- Verified everything in release helper
- Ran basic flows
- Saved/imported flows with NiFi Registry

On Fri, Jul 29, 2022 at 2:50 PM Joe Skora  wrote:
>
> +1 (binding) for release
>
> * helper guide review and build: hashes, certs, files checked out.
> * tested canvas and backend to create a test flow without any problems.

Re: [Nifi 1.16.1] Conflicting libraries when reading/writing to HDFS and Ozone

[Bryan Bende]

Yes creating a jira makes sense.

Likely someone needs to look at which version of Ozone client is being
used, and which version of HDFS client is being used, and then compare
all the transitive dependencies to see if there are any obvious
conflicts.

On Fri, Jul 29, 2022 at 8:54 AM Giorgio  wrote:
>
> Hello,
>
> Following up on this, I just wanted to ask if it’s suggested to open an issue.
>
> Thanks
>
> G
>
> Sent from Proton Mail for iOS
>
>
> On Mon, Jul 25, 2022 at 20:04, Giorgio  wrote:
>
> Hi Bryan,
>
> Thank you for getting back to me. Yes, that’s correct we compiled with the 
> ozone profile.
>
> G
>
> Sent from Proton Mail for iOS
>
>
> On Mon, Jul 25, 2022 at 18:12, Bryan Bende  wrote:
>
> Hello,
>
> Did you build your own NiFi off 1.16.1 using the profile
> "include-hadoop-ozone", and then used PutHDFS and that is what
> generated the error in your attached logs?
>
> -Bryan
>
> On Mon, Jul 25, 2022 at 11:08 AM Giorgio  wrote:
> >
> > Hello,
> >
> > I would like to report a potential bug that affects NiFi 1.16.1. Our 
> > current setup consists of a cluster of 3 nodes and the aim is to build a 
> > workflow that can write to both HDFS and Apache Ozone. To this end, we have 
> > configured two distinct PutHDFS processors. The configuration of these two 
> > processors is such that one points to HDFS and the other one to Ozone using 
> > two different core-sites configuration files. The PutHDFS processor that 
> > points to Ozone uses OFS as protocol.
> >
> > When we try running the workflow, however, we run into problems. 
> > Essentially, this seems to be caused by conflicting libraries that are 
> > instantiated by the two processors, thus generating an error.
> >
> > We have found a way to circumvent this problem but this required us to 
> > build a custom PutOzone processor that better encapsulates the way the 
> > libraries are instantiated.
> >
> > We are not sure whether this is a known issue. Please find attached the 
> > relevant log lines.
> >
> > Regards,
> >
> > Giorgio
> >
> >
> > Sent with Proton Mail secure email.

Re: OpenTelemetry Integration

[Bryan Bende]

Hi Greg,

I don't really know anything about OpenTelemetry, but from the
perspective of integrating something into the framework, some things
to consider...

Is there some way to piggy-back on provenance and use a ReportingTask
to process provenance events and report something to OpenTelemetry?

If something new does need to be added, it should probably be an
extension point where there is an interface in the framework-api and
different implementations can be plugged in.
Ideally the framework itself wouldn't have any knowledge of
OpenTelemetry specifically, it would only be reporting some
information, which could then be used in some way by the OpenTelemetry
implementation.

How does NiFi actually communicate with OpenTelemetry? Are you
expecting to send data to OpenTelemetry in this new method you are
suggesting?
That would likely have a significant impact on the performance of the flow.

Thanks,

Bryan

On Thu, Jul 28, 2022 at 3:17 PM glma...@uwe.nsa.gov  wrote:
>
> Nifi Devs,
>
> My team and I are looking for guidance on how we can extend Apache Nifi's 
> capabilities. Specifically we're looking to include distributed tracing. 
> We'll approach this effort as if we're the tracing experts and simply seeking 
> implementation guidance. Our developers have good exposure to working with 
> Nifi and creating custom processors. We plan to fork the project to begin 
> this effort but want to make sure we approach this with the best possible 
> direction for community adoption.
>
> Our initial thoughts on this approach would be to piggyback on how Provenance 
> was implemented. We essentially want to include a subroutine or method that 
> gets implicitly invoked upon a processors 'onTrigger' method. From there we 
> would analyze the FlowFiles attributes to check for the existence of 
> 'traceId' and/or propagate one if found.
>
> We can expound upon all of these tracing/observability details if that helps 
> by any means. We're able to provide more detailed scope of this task as well 
> but for now we just want to get feed back for our overall goal and proposed 
> approach.
>
> Thanks,
> Greg Marshall

Re: [Nifi 1.16.1] Conflicting libraries when reading/writing to HDFS and Ozone

[Bryan Bende]

Hello,

Did you build your own NiFi off 1.16.1 using the profile
"include-hadoop-ozone", and then used PutHDFS and that is what
generated the error in your attached logs?

-Bryan

On Mon, Jul 25, 2022 at 11:08 AM Giorgio  wrote:
>
> Hello,
>
> I would like to report a potential bug that affects NiFi 1.16.1. Our current 
> setup consists of a cluster of 3 nodes and the aim is to build a workflow 
> that can write to both HDFS and Apache Ozone. To this end, we have configured 
> two distinct PutHDFS processors. The configuration of these two processors is 
> such that one points to HDFS and the other one to Ozone using two different 
> core-sites configuration files. The PutHDFS processor that points to Ozone 
> uses OFS as protocol.
>
> When we try running the workflow, however, we run into problems. Essentially, 
> this seems to be caused by conflicting libraries that are instantiated by the 
> two processors, thus generating an error.
>
> We have found a way to circumvent this problem but this required us to build 
> a custom PutOzone processor that better encapsulates the way the libraries 
> are instantiated.
>
> We are not sure whether this is a known issue. Please find attached the 
> relevant log lines.
>
> Regards,
>
> Giorgio
>
>
> Sent with Proton Mail secure email.

[RESULT] [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.5

[Bryan Bende]

Apache NiFi Community,

I am pleased to announce that the Apache NiFi NAR Maven Plugin 1.3.5
release vote passes with:

  5 +1 (binding) votes
  3 +1 (non-binding) votes
  0 +0 votes
  0 -1 votes

Thanks to all who helped make this release possible!

Here is the vote thread:
https://lists.apache.org/thread/y47hzs4mvjzmspb3ol8gw7p9ymff49zx

Release Helper for Apache NiFi NAR Maven Plugin 1.3.5

[Bryan Bende]

Hello Apache NiFi community,

Please find the associated guidance to help those interested in
validating/verifying the release so they can vote.

# Download latest KEYS file:https://dist.apache.org/repos/dist/dev/nifi/KEYS

# Import keys file:
gpg --import KEYS

# [optional] Clear out local maven artifact repository

# Pull down nifi-nar-maven-plugin-1.3.5 source release artifacts for review:

wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/nifi-nar-maven-plugin-1.3.5-source-release.zip
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/nifi-nar-maven-plugin-1.3.5-source-release.zip.asc
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/nifi-nar-maven-plugin-1.3.5-source-release.zip.sha256
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/nifi-nar-maven-plugin-1.3.5-source-release.zip.sha512

# Verify the signature
gpg --verify -v nifi-nar-maven-plugin-1.3.5-source-release.zip.asc

# Verify the hashes (sha256, sha512) match the source and what was
provided in the vote email thread
shasum -a 256 nifi-nar-maven-plugin-1.3.5-source-release.zip
shasum -a 512 nifi-nar-maven-plugin-1.3.5-source-release.zip

# Unzip nifi-nar-maven-plugin-1.3.5-source-release.zip
unzip nifi-nar-maven-plugin-1.3.5-source-release.zip

# Verify the build works including release audit tool (RAT) checks
cd nifi-nar-maven-plugin-1.3.5
mvn clean install -Pcontrib-check

# Verify the contents contain a good README, NOTICE, and LICENSE.
# Verify the git commit ID is correct
# Verify the RC was branched off the correct git commit ID

# Verify that NiFi can build NARs correctly using the plugin

- Update NiFi's root pom to use version 1.3.5 of the
plugin
https://github.com/apache/nifi/blob/main/pom.xml#L102

- Perform a build of NiFi, optionally clear out local .m2 repo
mvn clean install

- Ensure that NiFi starts and loads all processors, controller
services, and reporting tasks

- Verify the overall runtime manifest correctly reflects provided APIs
with inheritance

Check 
nifi-manifest/nifi-runtime-manifest/target/classes/nifi-runtime-manifest.json
Look for AtomicDistributedMapCacheClient and any
providedApiImplementations that have this,
should also have DistributedMapCacheClient as a provided API since the
atomic API extends it.

# Send a response to the vote thread indicating a +1, 0, -1 based on
your findings.

Thank you for your time and effort to validate the release!

[VOTE] Release Apache NiFi NAR Maven Plugin 1.3.5

[Bryan Bende]

Hello Apache NiFi Community,

I am pleased to be calling this vote for the source release of Apache
NiFi NAR Maven Plugin 1.3.5.

The source being voted upon, including signatures, digests, etc. can
be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/

The Git tag is nifi-nar-maven-plugin-1.3.5-RC1
The Git commit ID is 77f516a65d796b715278bedd06407d4a8f98f771
https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=77f516a65d796b715278bedd06407d4a8f98f771

Checksums of nifi-nar-maven-plugin-1.3.5-source-release.zip:
SHA256: 073271bbd28b3d0b9b74c42ab889b4e7301e52b9a0579a3484e7ad0b7f8d3532
SHA512: 
9fcc487a197a8d3c5cd297696eb303015c9cde32f89541682dfac921e862c1f1ee8ba5e41573eeb7a5b5190b95c3b49443fca9ed5d5a3f7768a4ea1b031829af

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/bbende.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

2 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351862

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.5

The vote will be open for 24 hours. Please download the release
candidate and evaluate the necessary items including checking hashes,
signatures, build from source, and test. Then please vote:

[ ] +1 Release this package as nifi-nar-maven-plugin-1.3.5
[ ] +0 no opinion
[ ] -1 Do not release this package because ...

Re: [DISCUSS] Release for NAR Maven Plugin

[Bryan Bende]

Sounds good, I agree on going with a shorter voting period for such a
small release.

On Wed, Jul 20, 2022 at 3:43 PM Joe Witt  wrote:
>
> Bryan - yep sounds good.  For what it is worth I'd argue we just do a 24
> hour vote on this one...doesn't seem worth waiting 3 days.
>
> Thanks
>
> On Wed, Jul 20, 2022 at 12:38 PM Kevin Doran  wrote:
>
> >  +1 for cutting a release - thanks Bryan!
> >
> > On Jul 20, 2022 at 15:30:59, Bryan Bende  wrote:
> >
> > > If there are no objections, I'm planning to kick out another minor
> > > release for the NAR plugin to release a fix for a bug that was just
> > > resolved [1] .
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-10253
> > >
> >

[DISCUSS] Release for NAR Maven Plugin

[Bryan Bende]

If there are no objections, I'm planning to kick out another minor
release for the NAR plugin to release a fix for a bug that was just
resolved [1] .

[1] https://issues.apache.org/jira/browse/NIFI-10253

Re: generating authorizations on initial startup

[Bryan Bende]

I think you are right, it looks like in FlowParser parseJson, the
returned FlowInfo comes from this...

final VersionedProcessGroup rootGroup = dataflow.getRootGroup();
return new FlowInfo(rootGroup.getIdentifier(), ports);

Which I think should be the instance identifier there.

On Wed, Jul 13, 2022 at 11:57 AM Mark Bean  wrote:
>
> When starting NiFi for the first time using the managed-authorizer, NiFi
> will put the Initial Admin Identity in certain Access Policies. However, it
> only does this for Global Access Policies, and does not add this user to
> any Component Access Policies, e.g. 'view/modify the component'.
>
> This has been frustrating, but as I understand it is unavoidable because
> the UUID of the root process group has not yet been created (there is no
> flow.xml.gz) at the time the policies are generated.
>
> However, I found that if a flow.xml.gz existed without a corresponding
> authorizations.xml or users.xml, then the startup process would in fact
> create the Component Access Policies and add the admin user to them.
>
> Now, with the introduction of flow.json.gz, the root process group has
> both  "identifier" and "instanceIdentifier" properties. The Component
> Access Policies created on startup as described above reference the
> "identifier" UUID, but the UI indicates the "instanceIdentifier" is the
> proper UUID for the root process group. Therefore, the Component Access
> Policies are ineffective as they reference an incorrect UUID value.
>
> Is generating the Component Access Policies in this way supported?
>
> If so, then I will submit a ticket for using the proper UUID value when
> creating the policies.
>
> Thanks,
> Mark

Re: NiFi Registry - migrate from H2 to MariaDB

[Bryan Bende]

Hello,

What version of NiFi Registry are you trying to use with MariaDB?

There is a known issue in 1.16.0 that is addressed in 1.16.1:

https://issues.apache.org/jira/browse/NIFI-9950

Thanks,

Bryan


On Wed, May 11, 2022 at 2:44 PM John Wise  wrote:
>
> We'd like to move from the built-in H2 database to MariaDB, but the
> migration process doesn't appear to work when I specify the migration path
> & URL.  The Flyway scripts fail on the V2 sql script because it's claiming
> that MariaDB only supports 768 char columns, rather than the 1000 char
> columns in the script.  I can manually create the tables from the scripts,
> but then Flyway gets stuck in a migration loop on the next restart.
>
> We're using Git as the backend storage, and can recreate the H2 database
> from it at-will.  If I change to MariaDB and try that, it gets stuck on the
> Flyway errors again.
>
> The documentation on the site doesn't really go into a whole lot of detail
> on how to accomplish either method.  Is there a better way to accomplish
> this?
>
> Thanks!
> -John
>
> John Wise
> Principal Software Engineer
> Momentum Engineering

Re: Configurable default connection Prioritizer

[Bryan Bende]

Makes sense. For # 2, it is still per queue with an "Apply All"
convenience right? Just trying to differentiate with prioritizing
across all queues.

On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
 wrote:
>
> I see two things as particularly useful...
>
>  1) Default Prioritizer for new Relationships (Bound to a process group,
> similar to how the "Default FlowFile Expiration" can be changed).
>  2) Applying a prioritizer to an entire Process Group as a one-time action.
>
> Some background... I'm hand-converting two super-legacy v0.7.3 canvases to
> v1.15.3.  Part of that is applying flow priorities all over the place in
> the new system.  Probably not a common task, but I could see this feature
> being useful for other week-to-week work too.
>
> Ryan
>
> On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende  wrote:
>
> > I think there are two different concepts here... The original
> > discussion is just about default settings for new connections. The
> > idea in NIFI-6831 is about prioritizing data across multiple queues,
> > either for all of nifi or within a given process group.
> >
> >
> > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean  wrote:
> > >
> > > We experimented with the idea of a custom "Global Prioritizer". One of
> > the
> > > problems with this approach is that it ran the risk of breaking the
> > > multi-tenancy philosophy. If there were a truly global priority, it would
> > > affect all flows, each may have different priority rules. However, if
> > this
> > > could be applied only at the process group level, it might have legs.
> > >
> > > You can follow the initial approach to such a mechanism in the JIRA
> > ticket.
> > > https://issues.apache.org/jira/browse/NIFI-6831
> > >
> > >
> > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > ryan.andrew.hendrick...@gmail.com> wrote:
> > >
> > > > I just went to the config button in my process group, hoping to set all
> > > > relationships in there to priority first Lots of right clicking &
> > > > dragging instead.
> > > >
> > > > +1 for an approach like that.
> > > >
> > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt  wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Certainly the spirit of this is a good idea.  Would likely need to
> > > > approach
> > > > > it at a more flow/process group centric level.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > ryan.andrew.hendrick...@gmail.com> wrote:
> > > > >
> > > > > > This would be very helpful.
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > salvatoref...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Do you see much value in being able to specify an instance-wide
> > (or
> > > > > > > cluster-wide) default prioritizer for all connections that do not
> > > > have
> > > > > > one
> > > > > > > manually set?
> > > > > > >
> > > > > > > Along with the the following properties in nifi.properties:
> > > > > > > nifi.queue.backpressure.count=1
> > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > >
> > > > > > > I'd like to see see something like
> > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > PriorityAttributePrioritizer
> > > > > > >
> > > > > > > Thoughts? My only concern would be if connection prioritizers
> > have a
> > > > > > > noticeable impact on system resources.
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: Configurable default connection Prioritizer

[Bryan Bende]

I think there are two different concepts here... The original
discussion is just about default settings for new connections. The
idea in NIFI-6831 is about prioritizing data across multiple queues,
either for all of nifi or within a given process group.


On Tue, Apr 12, 2022 at 1:13 PM Mark Bean  wrote:
>
> We experimented with the idea of a custom "Global Prioritizer". One of the
> problems with this approach is that it ran the risk of breaking the
> multi-tenancy philosophy. If there were a truly global priority, it would
> affect all flows, each may have different priority rules. However, if this
> could be applied only at the process group level, it might have legs.
>
> You can follow the initial approach to such a mechanism in the JIRA ticket.
> https://issues.apache.org/jira/browse/NIFI-6831
>
>
> On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
>
> > I just went to the config button in my process group, hoping to set all
> > relationships in there to priority first Lots of right clicking &
> > dragging instead.
> >
> > +1 for an approach like that.
> >
> > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt  wrote:
> >
> > > Hello
> > >
> > > Certainly the spirit of this is a good idea.  Would likely need to
> > approach
> > > it at a more flow/process group centric level.
> > >
> > > Thanks
> > >
> > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > ryan.andrew.hendrick...@gmail.com> wrote:
> > >
> > > > This would be very helpful.
> > > >
> > > > Ryan
> > > >
> > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > salvatoref...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Do you see much value in being able to specify an instance-wide (or
> > > > > cluster-wide) default prioritizer for all connections that do not
> > have
> > > > one
> > > > > manually set?
> > > > >
> > > > > Along with the the following properties in nifi.properties:
> > > > > nifi.queue.backpressure.count=1
> > > > > nifi.queue.backpressure.size=1 GB
> > > > >
> > > > > I'd like to see see something like
> > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > PriorityAttributePrioritizer
> > > > >
> > > > > Thoughts? My only concern would be if connection prioritizers have a
> > > > > noticeable impact on system resources.
> > > > >
> > > >
> > >
> >

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

[Bryan Bende]

+1 (binding)

- Verified everything in release helper
- Ran a cluster and tested load-balanced connections, off-loading
data, and disconnecting/reconnecting nodes

On Thu, Mar 24, 2022 at 9:17 AM Robert Fellows  wrote:
>
> +1 (non-binding)
>
> Verified signature
> Verified hashes
> Build on MacOS Big Sur (11.6), JDK 11.0.8.hs-adpt
> Walked through a few simple use cases, all looks good.
>
> -- Rob
>
> On Thu, Mar 24, 2022 at 7:09 AM Tony Kurc  wrote:
>
> > +1 (binding)
> >
> > Verified hashes and signatures. Reviewed notice and license and readme and
> > did not see any issues. Build went without issue and ran a simple flow.
> >
> > Tony
> >
> >
> > On Mon, Mar 21, 2022, 5:42 PM Joe Witt  wrote:
> >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 1.16.0.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1198
> > >
> > > The source being voted upon and the convenience binaries can be found at:
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/
> > >
> > > A helpful reminder on how the release candidate verification process
> > works:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-1.16.0-RC3
> > > The Git commit ID is b019a9191f1c83bc7f547dc02c1b679b8936acee
> > >
> > >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b019a9191f1c83bc7f547dc02c1b679b8936acee
> > >
> > > Checksums of nifi-1.16.0-source-release.zip:
> > > SHA256: 2f16cb94df404d1bcc9c32835ba98da8940005a5d7ea5504c155ee42021a221e
> > > SHA512:
> > >
> > >
> > cbd95f15cec5ffe506fef204526267b18b77d7266f6dc3e1bbc3c7600aac12e119977f7d8cf93dbbbc86fbb0739ba88aaa11a5381d29a463ec9a0c9a18f4e9e6
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/joewitt.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 401 issues were closed/resolved for this release:
> > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350741
> > >
> > > Release note highlights can be found here:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test. Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.16.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> >
>
>
> --
> ---
> Rob Fellows

Re: nifi.apache.org website

[Bryan Bende]

I left a comment on the JIRA, but all NARs are released to Maven
central on each release. The profiles just determine what is included
in the assembly. So users can still get any of the non-included NARs
by downloading it from Maven central, it just isn't obvious how to
find it.

On Mon, Mar 21, 2022 at 8:18 AM Mike Thomsen  wrote:
>
> Pierre raised an interesting idea here:
> https://issues.apache.org/jira/browse/NIFI-7360
>
> We might want to consider a process for allowing a directory of
> downloadable NARs from third parties as a stop-gap while the Registry
> gets fleshed out on providing access to NARs.
>
> On Wed, Mar 16, 2022 at 5:04 PM Matt Gilman  wrote:
> >
> > Thanks Jon. The invite link for Slack is at the bottom of this page [1].
> >
> > [1] https://nifi.apache.org/mailing_lists.html
> >
> > On Wed, Mar 16, 2022 at 4:46 PM Jonathan Conti-Vock <
> > jonathan.contiv...@gmail.com> wrote:
> >
> > > Thanks for the update Steven. Correct me if I'm wrong, but I'll need
> > > someone to invite me as a guest to follow along in the slack channel,
> > > correct? I would be happy to and interested in lending my UI/UX experience
> > > to this conversation.
> > >
> > > Cheers,
> > >
> > > Jon
> > >
> > > On Wed, Mar 16, 2022 at 3:39 PM Steven Matison 
> > > wrote:
> > >
> > > > Just a quick update on this thread.  Progress has been made and we have
> > > an
> > > > initial hugo framework able to run the current site.
> > > >
> > > > Please join us in follow on conversation in #apachenifiwebsite on slack!
> > > >
> > > >
> > > >
> > > > On Fri, Mar 11, 2022 at 8:25 AM Joe Witt  wrote:
> > > >
> > > > > done.  and added you
> > > > >
> > > > > On Fri, Mar 11, 2022 at 6:17 AM Steven Matison <
> > > steven.mati...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > So glad there is such an interest in this.
> > > > > >
> > > > > > Can we move discussion into a slack channel dedicated to this topic?
> > > > > >
> > > > > > On Wed, Mar 9, 2022 at 2:47 PM Steven Matison <
> > > > steven.mati...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Sounds awesome, gimme some time to go through everything you guys
> > > > have
> > > > > > > shared.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 9, 2022 at 2:28 PM Andrew Lim <
> > > > andrewlim.apa...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Steven,
> > > > > > >>
> > > > > > >> Excited by your interest in this! A Jira was created to migrate
> > > the
> > > > > NiFi
> > > > > > >> website to ASF git build/deploy [1]. I’ve been meaning to work on
> > > it
> > > > > but
> > > > > > >> haven’t had the cycles.
> > > > > > >>
> > > > > > >> In my research on static-site generators, Hugo [2] came up as a
> > > very
> > > > > > >> strong candidate especially due to its speed. There are 10 Apache
> > > > > > websites
> > > > > > >> that currently use Hugo [3]. Apache Jena is one of them and that
> > > > team
> > > > > > >> documented their migration steps on their wiki [4].
> > > > > > >>
> > > > > > >> Perhaps you could do some research and share your findings. If we
> > > > can
> > > > > > get
> > > > > > >> a working version of the current NiFi website in a chosen
> > > generator,
> > > > > we
> > > > > > can
> > > > > > >> get a Discuss thread going in the community similar to what 
> > > > > > >> Apache
> > > > > Jena
> > > > > > did
> > > > > > >> [5].
> > > > > > >>
> > > > > > >> I would love to help with the migration as well as a refresh of
> > > the
> > > > > > site!
> > > > > > >>
> > > > > > >> -Drew
> > > > > > >>
> > > > > > >>
> > > > > > >> [1] https://issues.apache.org/jira/browse/NIFI-6848 <
> > > > > > >> https://issues.apache.org/jira/browse/NIFI-6848>
> > > > > > >> [2] https://gohugo.io/ 
> > > > > > >> [3]
> > > > > > >>
> > > > >
> > > https://github.com/search?q=topic%3Ahugo+org%3Aapache=Repositories
> > > > > > <
> > > > > > >>
> > > https://github.com/search?q=topic:hugo+org:apache=Repositories
> > > > >
> > > > > > >> [4]
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/INFRA/How+Apache+Jena+migrated+from+the+CMS
> > > > > > >> <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/INFRA/How+Apache+Jena+migrated+from+the+CMS
> > > > > > >> >
> > > > > > >> [5]
> > > > https://lists.apache.org/thread/tt4mm3phpkjxf7kc5hp3t006sob7ksqr
> > > > > <
> > > > > > >> https://lists.apache.org/thread/tt4mm3phpkjxf7kc5hp3t006sob7ksqr>
> > > > > > >>
> > > > > > >>
> > > > > > >> > On Mar 9, 2022, at 1:59 PM, David Handermann <
> > > > > > >> exceptionfact...@apache.org> wrote:
> > > > > > >> >
> > > > > > >> > Hi Steven,
> > > > > > >> >
> > > > > > >> > Thanks for your interest! This would also be a great 
> > > > > > >> > opportunity
> > > > to
> > > > > > >> > streamline the HTML generation approach to use something like
> > > Hugo
> > > > > or
> > > > > > >> > similar static-site generation 

Re: 1.16.0 RC1 checkstyle issue

[Bryan Bende]

Submitted a PR that should address the issue:

https://github.com/apache/nifi/pull/5861

On Fri, Mar 11, 2022 at 6:43 PM Mike Thomsen  wrote:
>
> Bryan,
>
> This is the dump of the file. I'm building on JDK 11 (Adoptium build)
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements.  See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License.  You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> Project-Version:1.16.0
> Build-Branch:
> Build-Revision:
> Build-Timestamp:${timestamp}
> Built-By:mikethomsen
> Maven-Home:/Users/mikethomsen/.sdkman/candidates/maven/current
> Maven-Version:3.6.3
> Created-By:Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Build-Java-Home:/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
> Build-Jdk:11.0.12
> Build-Jdk-Vendor:Eclipse Foundation
> Build-Arch:x86_64
> Build-Os:Mac OS X
> Build-Os-Version:11.2%
>
> On Fri, Mar 11, 2022 at 6:35 PM Nathan Gough  wrote:
> >
> > I also ran into both of these issues. Though it was just a me problem.
> >
> > The value in my build.properties is:
> > Build-Timestamp:${timestamp}
> >
> > On Fri, Mar 11, 2022 at 6:03 PM Bryan Bende  wrote:
> >
> > > Mike,
> > >
> > > What is the value of timestamp in your
> > > nifi-manifest/nifi-runtime-manifest/target/classes/build.properties ?
> > >
> > > On Fri, Mar 11, 2022 at 5:53 PM Mike Thomsen 
> > > wrote:
> > >
> > > > https://issues.apache.org/jira/browse/NIFI-9791
> > > >
> > > > On Fri, Mar 11, 2022 at 5:51 PM Mike Thomsen 
> > > > wrote:
> > > > >
> > > > > Will do
> > > > >
> > > > > On Fri, Mar 11, 2022 at 3:41 PM Joe Witt  wrote:
> > > > > >
> > > > > > just got word of a bug in some intensive testing so will be sinking
> > > the
> > > > > > release anyway.  Please file a JIRA for what you found.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Fri, Mar 11, 2022 at 1:27 PM Mike Thomsen  > > >
> > > > wrote:
> > > > > >
> > > > > > > I'm getting this build error locally in the nifi-runtime-manifest
> > > > > > > module. Going to abstain from voting for a little while to see if
> > > > > > > anyone has the bug since this machine is loaded up with stuff that
> > > > > > > sometimes makes builds break for me when they don't for others.
> > > Error
> > > > > > > is:
> > > > > > >
> > > > > > > [INFO] --- exec-maven-plugin:1.6.0:java
> > > (generate-runtime-manifest) @
> > > > > > > nifi-runtime-manifest ---
> > > > > > >
> > > > > > > Mar 11, 2022 3:24:56 PM
> > > > > > > org.apache.nifi.runtime.manifest.impl.RuntimeManifestGenerator 
> > > > > > > main
> > > > > > >
> > > > > > > INFO: Writing runtime manifest to:
> > > > > > >
> > > > > > >
> > > >
> > > /private/tmp/nifi-1.16.0/nifi-manifest/nifi-runtime-manifest/target/classes/nifi-runtime-manifest.json
> > > > > > >
> > > > > > > [WARNING]
> > > > > > >
> > > > > > > java.lang.NumberFormatException: For input string: "${timestamp}"
> > > > > > >
> > > > > > > at java.lang.NumberFormatException.forInputString
> > > > > > > (NumberFormatException.java:65)
> > > > > > >
> > > > > > > at java.lang.Long.parseLong (Long.java:678)
> > > > > > >
> > > > > > > at java.lang.Long.valueOf (Long.java:1144)
> > > > > > >
>

Re: 1.16.0 RC1 checkstyle issue

[Bryan Bende]

Mike,

What is the value of timestamp in your
nifi-manifest/nifi-runtime-manifest/target/classes/build.properties ?

On Fri, Mar 11, 2022 at 5:53 PM Mike Thomsen  wrote:

> https://issues.apache.org/jira/browse/NIFI-9791
>
> On Fri, Mar 11, 2022 at 5:51 PM Mike Thomsen 
> wrote:
> >
> > Will do
> >
> > On Fri, Mar 11, 2022 at 3:41 PM Joe Witt  wrote:
> > >
> > > just got word of a bug in some intensive testing so will be sinking the
> > > release anyway.  Please file a JIRA for what you found.
> > >
> > > Thanks
> > >
> > > On Fri, Mar 11, 2022 at 1:27 PM Mike Thomsen 
> wrote:
> > >
> > > > I'm getting this build error locally in the nifi-runtime-manifest
> > > > module. Going to abstain from voting for a little while to see if
> > > > anyone has the bug since this machine is loaded up with stuff that
> > > > sometimes makes builds break for me when they don't for others. Error
> > > > is:
> > > >
> > > > [INFO] --- exec-maven-plugin:1.6.0:java (generate-runtime-manifest) @
> > > > nifi-runtime-manifest ---
> > > >
> > > > Mar 11, 2022 3:24:56 PM
> > > > org.apache.nifi.runtime.manifest.impl.RuntimeManifestGenerator main
> > > >
> > > > INFO: Writing runtime manifest to:
> > > >
> > > >
> /private/tmp/nifi-1.16.0/nifi-manifest/nifi-runtime-manifest/target/classes/nifi-runtime-manifest.json
> > > >
> > > > [WARNING]
> > > >
> > > > java.lang.NumberFormatException: For input string: "${timestamp}"
> > > >
> > > > at java.lang.NumberFormatException.forInputString
> > > > (NumberFormatException.java:65)
> > > >
> > > > at java.lang.Long.parseLong (Long.java:678)
> > > >
> > > > at java.lang.Long.valueOf (Long.java:1144)
> > > >
> > > > at
> > > >
> org.apache.nifi.runtime.manifest.impl.RuntimeManifestGenerator.execute
> > > > (RuntimeManifestGenerator.java:81)
> > > >
> > > > at
> org.apache.nifi.runtime.manifest.impl.RuntimeManifestGenerator.main
> > > > (RuntimeManifestGenerator.java:143)
> > > >
> > > > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native
> > > > Method)
> > > >
> > > > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke
> > > > (NativeMethodAccessorImpl.java:62)
> > > >
> > > > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
> > > > (DelegatingMethodAccessorImpl.java:43)
> > > >
> > > > at java.lang.reflect.Method.invoke (Method.java:566)
> > > >
> > > > at org.codehaus.mojo.exec.ExecJavaMojo$1.run
> (ExecJavaMojo.java:282)
> > > >
> > > > at java.lang.Thread.run (Thread.java:829)
> > > >
> > > > On Fri, Mar 11, 2022 at 2:56 PM Joe Witt  wrote:
> > > > >
> > > > > Not from my point of view it isn't.  Also try it without the IT
> while
> > > > doing
> > > > > contrib-check then run the ITs without contrib-check.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Mar 11, 2022 at 12:51 PM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > [INFO]
> > > > > >
> > > > > > [INFO] --- maven-checkstyle-plugin:3.1.2:check (check-style) @
> > > > > > nifi-system-test-suite ---
> > > > > >
> > > > > > [WARNING]
> > > > > >
> > > >
> src/test/java/org/apache/nifi/tests/system/clustering/OffloadIT.java:[34,8]
> > > > > > (imports) UnusedImports: Unused import - java.util.Collection.
> > > > > >
> > > > > > [INFO]
> > > > > >
> > > >
> 
> > > > > >
> > > > > > [INFO] BUILD FAILURE
> > > > > >
> > > > > > [INFO]
> > > > > >
> > > >
> 
> > > > > >
> > > > > > [INFO] Total time:  15.602 s (Wall Clock)
> > > > > >
> > > > > > [INFO] Finished at: 2022-03-11T14:49:43-05:00
> > > > > >
> > > > > > [INFO]
> > > > > >
> > > >
> 
> > > > > >
> > > > > > [WARNING] The requested profile "include-grpc" could not be
> activated
> > > > > > because it does not exist.
> > > > > >
> > > > > > [ERROR] Failed to execute goal
> > > > > > org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:check
> > > > > > (check-style) on project nifi-system-test-suite: You have 1
> Checkstyle
> > > > > > violation. -> [Help 1]
> > > > > >
> > > > > > [ERROR]
> > > > > >
> > > > > >
> > > > > > Is this worth a -1 vote?
> > > > > >
> > > >
>

Re: Access to Community Slack

[Bryan Bende]

Hello,

We just updated the link on the site a few days ago.

Can you try the "invite" link here?

https://nifi.apache.org/mailing_lists.html

Thanks,

Bryan

On Thu, Feb 10, 2022 at 8:46 AM Nathaneal James  wrote:
>
> Hello,
>
> Is it possible to get an invite to the Apache Community Slack?
> I'm having trouble with Apache Nifi on Mac OSX.
>
>
> Thanks
> Nate

Re: Questions on creating NiFi custom UI

[Bryan Bende]

Hello,

Regarding bundling the WAR, take a look at the update attribute NAR.
If you look at the expanded NAR after starting NiFi, it is located at
"work/nar/extensions/nifi-update-attribute-nar-1.16.0-SNAPSHOT.nar-unpacked/NAR-INF/bundled-dependencies"
you will see inside there is a ".war", so you will want a similar
setup where you build a WAR for your custom UI and it ends up inside
your packaged NAR.

Regarding the Maven error, we would need to see a stacktrace for the
error. It may already be further up in the Maven output, or you could
run with -X to get much more detailed output.

Thanks,

Bryan

On Wed, Feb 9, 2022 at 8:24 AM Shun Jie Tan
 wrote:
>
> Hi,
>
> I have made some updates to my repository. Apparently I need to include the
> version for my dependencies, now 'mvn clean install' builds successfully
> for nifi-MapperUI-bundle and I copy-paste the nar file into my local niifi
> lib folder. But the advanced button doesn't appear when I create the
> MapperUI processor. What am I missing?
>
> As for nifi-MapperUIver1-bundle, there is a new error when I try to run
> 'mvn clean install'. This is the error it shows.
>
> [*ERROR*] Failed to execute goal
> org.apache.nifi:nifi-nar-maven-plugin:1.3.2:nar *(default-nar)* on project
> nifi-MapperUIver1-nar: *Failed to create Extension Documentation*: Failed
> to create Extension Definition for PROCESSOR
> org.apache.nifi.processors.MapperUIver1.MyProcessor -> *[Help 1]*
>
> I'm trying to figure out what is the cause of this error. Please advise me
> on how to fix this. Thank you.
>
>
> Best regards,
>
> Shun Jie
>
>
>
> On Wed, Feb 9, 2022 at 1:08 PM Shun Jie Tan 
> wrote:
>
> > Hello,
> >
> > I'm trying to create a custom processor with a custom UI that will appear
> > when the advanced button is clicked similar to UpdateAttribute and
> > JoltTransformJSON processors.
> >
> > The first step that I want to achieve is to have the advanced button
> > appear on my custom processor and display some simple text on the iframe
> > when clicked as a prototype. Using NiFi I installed locally to test out the
> > prototype for now.
> >
> > The current difficulty I'm facing is setting up the dynamic web project as
> > I'm not sure what is the right way to do it. Is there a maven archetype to
> > use for creating the custom UI?
> >
> > I have tried 2 ways to create my bundle.
> > First, I generated the nifi-processor-bundle-archetype using command line
> > and I created the dynamic web project separately using Eclipse IDE,
> > adding a pom.xml to it then copy-paste it into the
> > nifi-processor-bundle-archetype project bundle.
> >
> > For the second way, I created the nifi-processor-bundle-archetype project
> > in Eclipse IDE and added the dynamic web project directly into the project
> > bundle then, converting the dynamic web project into a maven project.
> >
> > Both of the projects produce 'jar is missing' error for some dependencies
> > when I run the 'mvn clean install' command. I had done 'maven > update
> > project' for the second project but the error still persists.
> >
> > I have seen the nifi documentation,
> > https://docs.cloudera.com/cfm/2.1.1/nifi-dev-guide/topics/nifi-custom-processor-uis.html,
> > for creating custom UI but I don't quite get the part where it says to
> > bundle the WAR file into a processor NAR.
> >
> > From NiFi's github repository, I see the structure for the
> > nifi-update-attribute-bundle contains the nifi-update-attribute-ui folder
> > so the bundling of the WAR file doesn't make sense to me.
> >
> > I have pushed my progress onto my github repository,
> > https://github.com/shunjie-t/nifi-custom-processor, for your reference.
> > Please guide me on how to set up the custom UI. Thank you.
> >
> > Best regards,
> > Shun Jie
> >

Re: NiFi Parameter Context Question

[Bryan Bende]

Hello,

1) I am not sure there is a definitive plan for removing variables
yet, but it is definitely something to consider for a NiFi 2.0 (or
other future release). I would also say that templates are in the same
situation.

Templates were created as a way for someone to share a part of a flow
with someone else, not really as a deployment mechanism. We have now
centered around the "flow definition" JSON used by NiFi Registry as
the format that should be used for importing/exporting/deploying. You
can right-click on a PG and Download Flow Definition, or create a new
PG and upload the flow definition JSON, or save/import to/from a NiFi
Registry. If you use this format, it does in fact capture the
parameter contexts and their initial values, minus sensitive values,
and it will create the contexts on import to a new NiFi instance.

2) This has come up a few times... it was an intentional decision to
not allow a sub-PG to inherit the param context from a parent PG, as
it becomes complicated to understand what value is actually going to
be resolved. However, I do think we need a user-friendly way to allow
setting the context on all sub-PGs. So if you were to configure a
top-level process group and set the param context, you would have an
option to select that all sub-PGs are also set to this. It is unclear
if/when this would be implemented, as someone would have to take on
the work to implement it.

Thanks,

Bryan

On Wed, Jan 26, 2022 at 11:05 AM Fredrick Pletz II
 wrote:
>
> Good morning,
> I love the introduction of parameters.  They satisfy a longstanding need for 
> managing multiple sensitive values from a single spot.  However, the 
> implementation does not satisfy the same functionality as variables.  Based 
> on the comment inside NiFi stating that parameters are replacing variables, 
> this has me worried that I'm going to lose significant functionality if 
> variables stop being supported.  Specifically, I use variables to assign 
> non-sensitive values throughout a NiFi in a way that can be transferred via 
> templates.  This allows me to create a generic template, deploy that 
> template, and configure the specifics by just setting a few variables.  It 
> has sped up deployment immensely.  Parameters don't transfer with templates, 
> and have to be added to every processing group.  I don't know of a way to 
> have all processing groups set to a parameter context all at once, so I would 
> end up needing to create the parameter context, add all the values, then add 
> that parameter context to dozens of processing groups.  This would be a major 
> step backward in streamlining deployments.
> Question 1) is NiFi planning to abandon support for variables?
> Question 2) is there a way to address the issues I have with parameter 
> contexts? - inability to assign a parameter context to ALL processing 
> groups, or even all sub-processing groups under a selected processing group   
>   - inability to transfer parameter context to new NiFi instances
> Fred Pletz

Re: Backfilling in Nfi

[Bryan Bende]

Hello,

The GetSplunk processor has 3 time range strategies...

- Managed from beginning
- Managed from current
- Provided

The "managed" strategies store the last successful time range in
processor state, so when splunk is down it should keep trying to query
Splunk and failing and never updating the state, so each next
execution would still have the start time as the end of the last
success, and therefore would capture the whole range that Splunk was
down.

It looks like you may be using the "provided" strategy since you are
specifying values for Earliest and Latest. The provided strategy was
meant to just query exactly what is provided, so it does not store
state or create incremental time ranges automatically, which means you
won't get the backfill in this case.

Thanks,

Bryan


On Wed, Jan 12, 2022 at 4:53 PM Chirthani, Deepak Reddy
 wrote:
>
> Hey Mark,
>
> Good to see you. I'm specifically talking about GetSplunk Processor which we 
> use to pull data previous 15 minutes data(earliest time: -15m@m and latest 
> time: now()) from the Splunk Enterprise every 15 mins. The processor 
> absolutely works fine. But it is Splunk with which we have issues of getting 
> down so that we miss data when it is down. So trying to find a way of 
> backfilling the data. Does this makes sense? Let me know if you have further 
> questions.
>
>
>
> On 2022/01/11 22:36:39 "Chirthani, Deepak Reddy" wrote:
>
> > Hello,
>
> >
>
> > Is Nifi good at Backfilling the data? Let's say that I have a dataflow 
> > which is cron scheduled to run every 15 mins. Lets say that the source was 
> > down and the dataflow did not pulled any data for a couple of runs and 
> > after some time the source is up and running. If this happens frequently, 
> > how do you handle backfilling in Nifi? Please let me know if there is any 
> > automated way of doing this?
>
> >
>
> > Thanks
>
> > E-MAIL CONFIDENTIALITY NOTICE:
>
> > The contents of this e-mail message and any attachments are intended solely 
> > for the addressee(s) and may contain confidential and/or legally privileged 
> > information. If you are not the intended recipient of this message or if 
> > this message has been addressed to you in error, please immediately alert 
> > the sender by reply e-mail and then delete this message and any 
> > attachments. If you are not the intended recipient, you are notified that 
> > any use, dissemination, distribution, copying, or storage of this message 
> > or any attachment is strictly prohibited.
>
> >
> E-MAIL CONFIDENTIALITY NOTICE:
> The contents of this e-mail message and any attachments are intended solely 
> for the addressee(s) and may contain confidential and/or legally privileged 
> information. If you are not the intended recipient of this message or if this 
> message has been addressed to you in error, please immediately alert the 
> sender by reply e-mail and then delete this message and any attachments. If 
> you are not the intended recipient, you are notified that any use, 
> dissemination, distribution, copying, or storage of this message or any 
> attachment is strictly prohibited.

Re: Configuring NiFi To Read From LDAP Groups

[Bryan Bende]

The admin guide should cover most of the scenarios:

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referencing-user-dn

On Tue, Dec 14, 2021 at 12:53 PM Michael Radov (RIT Alumni)
 wrote:
>
> Bryan,
>
> Thank you very much! After setting the ldap-provider information in the 
> logion-identity-providers.xml and the LDAP Group Provider in the 
> authorizers.xml, one would use the authorizers.xml file to load the user 
> groups. There wont be a need to use the files to be a User Group Provider if 
> I am using just LDAP. Is there a guide that one can use for creating 
> file-based policy provider information in the authorizations.xml file?
>
> Best,
> Mike R
>
> On Tue, Dec 14, 2021 at 12:14 PM Bryan Bende  wrote:
>>
>> CAUTION: This message is from an off campus source. Access to web links will 
>> be filtered (by proxy) for additional protection. Click with caution.
>> CAUTION: This message came from outside RIT. If you are unsure about the 
>> source or content of this message, please contact the RIT Service Center at 
>> 585-475-5000 or help.rit.edu before clicking links, opening attachments or 
>> responding.
>>
>>
>> Hello,
>>
>> The standard authorizer is composed of a user-group-provider and a
>> policy-provider. The LDAP user-group-provider can be be used to load
>> groups from LDAP, but you still need to define policies on them which
>> would be through a policy-provider, most likely the File-based
>> policy-provider which stores policies in authorizations.xml.
>>
>> Thanks,
>>
>> Bryan
>>
>> On Tue, Dec 14, 2021 at 10:02 AM Michael Radov (RIT Alumni)
>>  wrote:
>> >
>> > Hey,
>> >
>> > I am looking to see if there is a way to get NiFi to read directly from
>> > LDAP Groups. If this were the case, I would use the User Groupo Provider to
>> > ldap-user-group-provider. However, would one still need to use an
>> > authorizations file?
>> >
>> > I was reading through the work that Pierre Villard did on LDAP Group
>> > Authentication and authorization
>> > <https://pierrevillard.com/2017/12/22/authorizations-with-ldap-synchronization-in-apache-nifi-1-4/>
>> > using
>> > NiFi in a similar way and wanted to know if the file user group provider
>> > was necessary?
>> >
>> > Best,
>> > Mike R

Re: Configuring NiFi To Read From LDAP Groups

[Bryan Bende]

Hello,

The standard authorizer is composed of a user-group-provider and a
policy-provider. The LDAP user-group-provider can be be used to load
groups from LDAP, but you still need to define policies on them which
would be through a policy-provider, most likely the File-based
policy-provider which stores policies in authorizations.xml.

Thanks,

Bryan

On Tue, Dec 14, 2021 at 10:02 AM Michael Radov (RIT Alumni)
 wrote:
>
> Hey,
>
> I am looking to see if there is a way to get NiFi to read directly from
> LDAP Groups. If this were the case, I would use the User Groupo Provider to
> ldap-user-group-provider. However, would one still need to use an
> authorizations file?
>
> I was reading through the work that Pierre Villard did on LDAP Group
> Authentication and authorization
> 
> using
> NiFi in a similar way and wanted to know if the file user group provider
> was necessary?
>
> Best,
> Mike R

[RESULT] [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

[Bryan Bende]

I am pleased to announce that the 1.3.3 release of Apache NiFi NAR
Maven Plugin passes with
7 +1 (binding) votes
1 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:

https://mail-archives.apache.org/mod_mbox/nifi-dev/202111.mbox/%3CCALo_M198g0q1UmWy-udXKqABNuXzp-2rrt0B9-o_Ukmg9WxrQA%40mail.gmail.com%3E

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

[Bryan Bende]

+1 (binding)

On Fri, Dec 3, 2021 at 12:12 PM Kevin Doran  wrote:
>
> +1, binding
>
> Verified sigs/hashes/license/notice. Tried it out and it works as described. 
> Thanks Bryan!
>
> > On Dec 2, 2021, at 12:09, Matt Gilman  wrote:
> >
> > +1 (binding)
> >
> > Ran through the release helper. Looked great! Thanks for RMing Bryan!
> >
> > Matt
> >
> > On Thu, Dec 2, 2021 at 11:28 AM Matt Burgess  wrote:
> >
> >> +1 (binding), ran through release helper and verified NARs were loaded
> >> successfully into a NiFi instance.  Thanks for RM'ing Bryan!
> >>
> >> On Tue, Nov 30, 2021 at 3:56 PM Bryan Bende  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am pleased to be calling this vote for the source release of Apache
> >>> NiFi NAR Maven Plugin 1.3.3.
> >>>
> >>> The source zip, including signatures, digests, etc. can be found at:
> >>> https://repository.apache.org/content/repositories/orgapachenifi-1188
> >>>
> >>> The source being voted upon and the convenience binaries can be found at:
> >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/
> >>>
> >>> A helpful reminder on how the release candidate verification process
> >> works:
> >>>
> >> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >>>
> >>> The Git tag is nifi-nar-maven-plugin-1.3.3-RC1
> >>> The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
> >>>
> >> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
> >>>
> >>> Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip:
> >>> SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77
> >>> SHA512:
> >> 0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe
> >>>
> >>> Release artifacts are signed with the following key:
> >>> https://people.apache.org/keys/committer/bbende.asc
> >>>
> >>> KEYS file available here:
> >>> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>>
> >>> 1 issue was closed/resolved for this release:
> >>>
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348815
> >>>
> >>> Release note highlights can be found here:
> >>>
> >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3
> >>>
> >>> The vote will be open for 72 hours. Please download the release
> >>> candidate and evaluate the necessary items including checking hashes,
> >>> signatures, build from source, and test. Then please vote:
> >>>
> >>> [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3
> >>> [ ] +0 no opinion
> >>> [ ] -1 Do not release this package because...
> >>
>

Release helper for Apache NiFi NAR Maven Plugin 1.3.3

[Bryan Bende]

Hello,

The standard release candidate verification helper can be found here:

https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

In addition, the following steps can be used to verify the improvement
made in this release...

1) Download the source from:

https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/nifi-nar-maven-plugin-1.3.3-source-release.zip

2) Unzip the source and run "mvn clean install"

3) In your NiFi source code, edit the root pom.xml to change the
version of the NAR plugin from 1.3.2 to 1.3.3

4) Run a full build of NiFi, "mvn clean install" or whatever you prefer

5) Locate an extension-manifest.xml from one of the NARs, example:

ls -l nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-nar/target/META-INF/docs/
Nov 30 14:35 additional-details
Nov 30 14:35 extension-manifest.xml

6) Verify that the extension-manifest.xml now contains the groupId,
artifactId, version, parentNar, and buildInfo. You may want to open
the file in an editor that can pretty print the XML, in Intellij this
is the "Code -> Reformat Code" option.


org.apache.nifi
nifi-amqp-nar
1.16.0-SNAPSHOT

org.apache.nifi
nifi-standard-services-api-nar
1.16.0-SNAPSHOT

1.16.0-SNAPSHOT

nifi-1.15.0-RC3
nar-plugin-test
3e095ab
1.8.0_282
bbende
2021-11-30T14:35:04Z

[VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

[Bryan Bende]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi NAR Maven Plugin 1.3.3.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1188

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-nar-maven-plugin-1.3.3-RC1
The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2

Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip:
SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77
SHA512: 
0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/bbende.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

1 issue was closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348815

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3

The vote will be open for 72 hours. Please download the release
candidate and evaluate the necessary items including checking hashes,
signatures, build from source, and test. Then please vote:

[ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3
[ ] +0 no opinion
[ ] -1 Do not release this package because...

[discuss] Release of NAR plugin

[Bryan Bende]

All,

I'd like to kick out a release of the NAR plugin to make a minor
improvement [1] available for the extension-manifest XML that is
generated during the build.

I can take care of putting together the release.

Thanks,

Bryan

[1] https://issues.apache.org/jira/browse/NIFI-9404

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

[Bryan Bende]

+1 (binding)

Verified everything in the release helper and tested saving/importing
flows to/from registry with inherited param contexts.

On Thu, Nov 4, 2021 at 5:56 AM Kotaro Terada  wrote:
>
> +1 (non-binding)
>
> - Verified hashes.
> - Built from source with OpenJDK 8 and OpenJDK 11.
> - Run a 3-node secure cluster.
> - Tested a couple of flows.
> - Tested new features (ExecuteStateless, inherited parameter context, etc).
>
> Thank you for working on this, Joe.
>
> Thanks,
> Kotaro
>
>
> On Thu, Nov 4, 2021 at 10:44 AM Otto Fowler  wrote:
>
> >  +1
> > Ran through verification, build and run
> >
> > From: Joe Witt  
> > Reply: dev@nifi.apache.org  
> > Date: November 3, 2021 at 15:29:41
> > To: dev@nifi.apache.org  
> > Subject:  [VOTE] Release Apache NiFi 1.15.0 (rc3)
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 1.15.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1186
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
> >
> > A helpful reminder on how the release candidate verification process works:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.15.0-RC3
> > The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b
> >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b
> >
> > Checksums of nifi-1.15.0-source-release.zip:
> > SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a
> > SHA512:
> >
> > 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f
> >
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 234 issues were closed/resolved for this release:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382
> >
> > Release note highlights can be found here:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.15.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >

[ANNOUNCE] New NiFi PMC Member David Handermann

[Bryan Bende]

NiFi Community,

On behalf of the Apache NiFI PMC, I am very pleased to announce that
David Handermann has accepted the PMC's invitation to join the Apache
NiFi PMC.

David has made significant improvements in a number of security
related areas, he continues to improve the stability of our tests and
CI builds, and regularly helps out the community through the mailing
lists, Slack, and PR reviews.

Please join us in congratulating and welcoming David to the Apache NiFi PMC.

Congratulations David!

Re: How to Edit DBCPConnectionPoolLookup Through CLI

[Bryan Bende]

Hello,

The CLI doesn't currently have a command to set a property on a processor.
It would be a good one to add though, something like "proc-set-prop".

You could also figure out the call being made by performing the action in
the UI and looking at Chrome Dev Tools to understand the calls being made,
and then automate the call using curl.

Thanks,

Bryan




On Sat, Aug 21, 2021 at 9:06 AM Akın Kurtulan
 wrote:

> Hello,
>
>
>
> We are planning to use nifi in our systems and want some things automated
> using nfi toolkit.
>
> We are able to create controller services (DBCPConnectionPool 1.12.1)
> with  toolkit.
>
> Our next move is to add the created service to a DBCPConnectionPoolLookup
> 1.12.1 controller service.
>
> But i can not find a way to do it from comand line.
>
>
>
> Is there a way to do it without entering GUI?
>
>
>
> Regards,
>
>
> Akın Kurtulan
> Bilgi Teknolojileri Bölümü
> *Tel:  * 0216 552 5126
> *Faks:*0212 316 0836
>
>
>
> --
>
> Bu e-posta mesajı ve ekleri gönderildiği kişi ya da kuruma özeldir ve
> gizlidir. Ayrıca hukuken de gizli olabilir. Hiçbir şekilde üçüncü kişilere
> açıklanamaz ve yayınlanamaz. Mesajın yetkili alıcısı değilseniz hiçbir
> kısmını kopyalayamaz, başkasına gönderemez veya hiçbir şekilde
> kullanamazsınız. Eğer mesajın yetkili alıcısı veya yetkili alıcısına
> iletmekten sorumlu kişi siz değilseniz, lütfen mesajı sisteminizden siliniz
> ve göndereni uyarınız. Gönderen ve TÜRKİYE İŞ BANKASI A.Ş. bu mesajın
> içerdiği bilgilerin doğruluğu, bütünlüğü ve güncelliği konusunda bir
> garanti vermemektedir. Mesajın içeriğinden, iletilmesinden, alınmasından,
> saklanmasından, gizliliğinin korunamamasından, virüs içermesinden ve
> sisteminizde yaratabileceği zararlardan Bankamız sorumlu tutulamaz.
> Bankamıza gönderdiğiniz e-posta mesajlarında ve/veya eklerinde yer
> alan/alabilecek fikir, düşünce, tasarım, proje, sunum vb. içeriklerin ise
> gizli olmadığı, bu içerikleri serbest iradenizle paylaşmış olduğunuz kabul
> edilmektedir. Gönderdiğiniz e-posta mesajlarında ve/veya eklerinde yer alan
> içerikleri kullanıp kullanmama Bankamızın tek taraflı takdir yetkisindedir.
> Bankamızın bu içerikleri koruma, saklama, üçüncü şahıslara açıklamama gibi
> yükümlülüğü ve bu içeriklerin Bankamız nezdinde yapılmakta olan çalışmalar
> neticesinde meydana getirilmiş/getirilecek fikri ve sınai mülkiyet hakkı
> konusu oluşturan veya oluşturmayan Bankamız ürünlerine benzediği, bu
> içeriklerden esinlenildiği gibi iddialar da dahil olmak üzere buna benzer
> talepleriniz karşısında Bankamızın herhangi bir hukuki/cezai sorumluluğu
> bulunmamaktadır. Bankamıza gönderdiğiniz e-posta mesajlarında ve/veya
> eklerinde yer alan/alabilecek içeriklerin gizli ve/veya fikri, sınai
> mülkiyet hakkı kapsamında olduğunu düşünüyor iseniz talep edilmedikçe bu
> tür içerikleri Bankamız ile paylaşmayınız. Gönderdiğiniz e-posta
> mesajlarında ve/veya eklerinde yer alan içeriklerin üçüncü şahısların
> haklarını kısmen veya tamamen ihlal etmesi halinde, Bankamızın bu konuda
> ortaya çıkabilecek ihtilafların tarafı olmayacağını ve sorumluluğun tamamen
> tarafınıza ait olacağını bildiririz.
> Türkiye İş Bankası A.Ş. Mersis No:0481005859000909
>
> This e-mail and its attachments are private and confidential to the
> exclusive use of the individual or entity to whom it is addressed. It may
> also be legally confidential. Any disclosure, distribution or other
> dissemination of this message to any third party is strictly prohibited. If
> you are not the intended recipient, you may not copy, forward, send or use
> any part of it. If you are not the intended recipient or the person who is
> responsible to transmit to the intended recipient, please contact the
> sender by reply e-mail and destroy all copies of the original message and
> its attachments. The sender and TURKIYE IS BANKASI A.S. do not warrant for
> the accuracy, currency, integrity or correctness of the information in the
> message and its attachments. TURKIYE IS BANKASI A.S. shall have no
> liability with regard to the information contained in the message, its
> transmission, reception, storage, preservation of confidentiality, viruses
> or any damages caused in anyway to your computer system. The ideas,
> thoughts, designs, projects, presentations and the like contents, which may
> be contained within the email messages sent by you to our Bank and/or the
> attachments of the same (“Contents”), are assumed not to be confidential,
> and are shared by you, acting on your free will. It is up to the sole
> discretion of the Bank to or not to use the Contents. Our Bank is not
> obliged to protect and preserve contents or to abstain from the disclosure
> of the same to third parties, and does not assume any legal/criminal
> liability in respect of any claims you may file, including but not limited
> to those that could be based on the allegations that Contents are similar
> to or have been inspired from through the course of the 

Re: [DISCUSS] NiFi 2.0 Release Goals

[Bryan Bende]

> Java
> > > 8,
> > > > > and finally start on 2.x which would support only Java 11. I'm unsure
> > > of
> > > > > what implications changing the date and time handling would have -
> > for
> > > > > running systems that use long term historical logs, unexpected
> > impacts
> > > to
> > > > > time logging could be a problem.
> > > > >
> > > > > As Joe says I think feature work will have to be dedicated to 2.x and
> > > we
> > > > > could support 1.x for security fixes for some period of time. 2.x
> > seems
> > > > > like a gargantuan task but it's probably time to get started. Not
> > sure
> > > > how
> > > > > we handle all open PRs and the transition between 1.x and 2.x.
> > > > >
> > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt  >  > > > > joe.w...@gmail.com>> wrote:
> > > > >
> > > > > Jon
> > > > >
> > > > > You're right we have to be careful and you're right there are still
> > > > > significant Java 8 users out there. But we also have to be careful
> > > > > about security and sustainability of the codebase. If we had talked
> > > > > about this last year when that article came out I'd have agreed it is
> > > > > too early. Interestingly that link seems to get updated and I tried
> > > > > [1] and found more recent data (not sure how recent). Anyway it
> > > > > suggests Java 8 is still the top dog but we see good growth on 11. In
> > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem
> > > > > to care about Java 11 until later half last year and now suddenly it
> > > > > is all over the place.
> > > > >
> > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> > > > > work on the 1.x line just being blunt. We did this many years ago
> > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> > > > > so) but it was purely bug fix/security related bits. We would need to
> > > > > do something similar. But feature work would almost certainly go to
> > > > > the 2.x line. Maybe there are other workable models but my instinct
> > > > > suggests this is likely to follow a similar path.
> > > > >
> > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We
> > > > > need to make the call in both the interests of the user base and the
> > > > > contributor base of the community.
> > > > >
> > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> > > > >
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt  > > > > joe.w...@gmail.com>> wrote:
> > > > > Russ
> > > > >
> > > > > Yeah the flow registry is a key part of it. But also now you can
> > > > > download the flow definition in JSON (upload i think is there now
> > > > > too). Templates offered a series of challenges such as we store them
> > > > > in the flow definition which has made flows massive in an unintended
> > > > > way which isn't fun for cluster behavior.
> > > > >
> > > > > We have a couple cases where we headed down a particular concept and
> > > > > came up with better approaches later. We need to reconcile these with
> > > > > the benefit of hindsight, and while being careful to be not overly
> > > > > disruptive to existing users, to reduce the codebase/maintenance
> > > > > burden and allow continued evolution of the project.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman <
> > r...@windofkeltia.com
> > > > > <mailto:r...@windofkeltia.com>>
> > > > > wrote:
> > > > > Joe,
> > > > >
> > > > > I apologize for the off-topic intrusion, but what replaces templates?
> > > > > The Registry? Templates rocked and we have used them since 0.5.x.
> > > > >
> > > > > Russ
> > > > >
> > > > > On 7/23/21 8:31 AM, Joe Witt wrote:
> > > > > David,
> > > > >
&g

Re: [DISCUSS] NiFi 2.0 Release Goals

[Bryan Bende]

I'm a +1 for this... Not sure if this falls under "Removing Deprecated
Components", but I think we should also look at anything that has been
marked as deprecated throughout the code base as a candidate for
removal. There are quite a few classes, methods, properties, etc that
have been waiting for a chance to be removed.

On Fri, Jul 23, 2021 at 10:13 AM David Handermann
 wrote:
>
> Team,
>
> With all of the excellent work that many have contributed to NiFi over the
> years, the code base has also accumulated some amount of technical debt. A
> handful of components have been marked as deprecated, and some components
> remain in the code base to support integration with old versions of various
> products. Following the principles of semantic versioning, introducing a
> major release would provide the opportunity to remove these deprecated and
> unsupported components.
>
> Rather than focusing the next major release on new features, what do you
> think about focusing on technical debt removal? This approach would not
> make for the most interesting release, but it provides the opportunity to
> clean up elements that involve breaking changes.
>
> Focusing on technical debt, at least three primary goals come to mind for
> the next major release:
>
> 1. Removal of deprecated and unmaintained components
> 2. Require Java 11 as the minimum supported version
> 3. Transition internal date and time handling to JSR 310 java.time
> components
>
> *Removing Deprecated Components*
>
> Removing support for older and deprecated components provides a great
> opportunity to improve the overall security posture when it comes to
> maintaining dependencies. The OWASP dependency plugin report currently
> generates 50 MB of HTML for questionable dependencies, many of which are
> related to old versions of various libraries.
>
> As a starting point, here are a handful of components and extension modules
> that could be targeted for removal in a major version:
>
> - PostHTTP and GetHTTP
> - ListenLumberjack and the entire nifi-lumberjack-bundle
> - ListenBeats and the entire nifi-beats-bundle
> - Elasticsearch 5 components
> - Hive 1 and 2 components
>
> *Requiring Java 11*
>
> Java 8 is now over seven years old, and NiFi has supported general
> compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> internal improvements specifically related to TLS 1.3, which allowed
> closing out the long-running Java 11 compatibility epic NIFI-5174. Making
> Java 11 the minimum required version provides the opportunity to address
> any lingering edge cases and put NiFi in a better position to support
> current Java versions.
>
> *JSR 310 for Date and Time Handling*
>
> Without making the scope too broad, transitioning internal date and time
> handling to use DateTimeFormatter instead of SimpleDateFormat would provide
> a number of advantages. The Java Time components provide much better
> clarity when it comes to handling localized date and time representations,
> and also avoid the inherent confusion of java.sql.Date extending
> java.util.Date. Many internal components, specifically Record-oriented
> processors and services, rely on date parsing, leading to confusion and
> various workarounds. The pattern formats of SimpleDateFormat and
> DateTimeFormatter are very similar, but there are a few subtle differences.
> Making this transition would provide a much better foundation going forward.
>
> *Conclusion*
>
> Thanks for giving this proposal some consideration. Many of you have been
> developing NiFi for years and I look forward to your feedback. I would be
> glad to put together a more formalized recommendation on Confluence and
> write up Jira epics if this general approach sounds agreeable to the
> community.
>
> Regards,
> David Handermann

Re: [DISCUSS] NiFi Registry -> NiFi consensus

[Bryan Bende]

+1

On Fri, Jul 16, 2021 at 2:31 PM Kevin Doran  wrote:
>
> +1.
>
> > On Jul 16, 2021, at 2:28 PM, Joe Witt  wrote:
> >
> > Yep definitely. Thought this was sorted via the JIRA.
> >
> > +1
> >
> > On Fri, Jul 16, 2021 at 11:26 AM David Handermann
> >  wrote:
> >>
> >> Thanks for handling this, Matt. As one of the reviewers on the PR, I vote
> >> +1 (non-binding).
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Fri, Jul 16, 2021 at 1:20 PM Matt Burgess  wrote:
> >>
> >>> All,
> >>>
> >>> We've been asked to record our consensus for the move of NiFi Registry
> >>> to the NiFi codebase in a mailing list thread for posterity. Most
> >>> discussion happened on the PR but INFRA would like a link to this
> >>> thread showing consensus from PMC members, committers, and the
> >>> community.
> >>>
> >>> I didn't put my +1 on the PR but I am in favor of moving the NiFi
> >>> Registry codebase into NiFi :) Please feel free to share your thoughts
> >>> as well.
> >>>
> >>> Thank you,
> >>> Matt
> >>>
>

Re: Apache NiFi Registry

[Bryan Bende]

It was done as a precaution that a user may not realize that after
import or change-version, that the entire flow is going to
automatically start running. For cases where there is an existing flow
and the next version maybe adds one processor in the middle, it
probably wouldn't be a big deal, but that can't easily be
differentiated from a whole new section of the flow that starts
pulling data from an external system in production. We could probably
add an option to make it easier, but I think most people are
automating this in some way and it ends up being two API calls to
enable valid services and start valid processors.

On Tue, Jul 6, 2021 at 2:45 PM Phillip Lord  wrote:
>
> Thanks Andrew,
>
> Yes we generally do not leave any components in a "stopped" state.  They're 
> always "running/enabled/disabled".
> What we're hoping to do is utilize a "staging" environment for all changes as 
> opposed to having to have any direct manipulation to the production clusters. 
>  Where the staging environment would then check-in changes to the registry... 
> The production environment then recognized the pg isn't utilizing the newest 
> available version and we can then tell the production environment to check in 
> the latest version.  As it stands today if the change that I checked in is aa 
> new component in a flow, the new component that the production environment 
> checks-in is in a stopped state.  This then requires additional manual 
> intervention to "enable/start" the flow.
>
> I can see it potentially being a precautionary type thing... but would 
> definitely be useful to have the option to have components retain 
> status/state when checked into the registry.
>
> I imagined it would be somewhat similar to how MiNiFi works where components 
> are assumed to be running at all times... at least I think that's how it's 
> working in the background?
>
>
>
> On 2021/07/01 21:34:52, Andrew Grande  wrote:
> > Isn't the proper state for this use case enabled/disabled? NiFi will start
> > a PG and every schedulable component in it. If one needs to prevent this,
> > disable a processor.
> >
> > Andrew
> >
> > On Thu, Jul 1, 2021, 2:17 PM Phillip Lord 
> > wrote:
> >
> > > Hello,
> > >
> > > My organization is considering utilizing the Registry.  From my testing it
> > > appears that versioning doesn't keep track of the state of components
> > > (stopped/started/etc).  Is this accurate?  Are there plans to have
> > > versioning keep track of this in future releases?
> > >
> > > I'm using NiFi 1.11.4 and Registry version 0.8.0
> > >
> > > Thanks,
> > > Phil Lord
> > >
> >

Re: Issues with NiFi, NiFi Registry and GIT

[Bryan Bende]

Hello,

1) The folder structure in git is controlled by NiFi registry, so it
is not possible to create any different structure than what you are
seeing. Think of it like an internal database, you shouldn't really
need to know/care about the structure.

2a) Starting with NiFI 1.10.0 and registry 0.5.0, the disabled state
is captured when saving a versioned flow and will remain when imported
to another environment, so this should already be working.

2b) This is already how it works, it is a manual process to start
everything in the process group after initial import.

2c) This one I would have to double check, but I would expect it to
work as you described, meaning if something was stopped in the target
env, it would remain stopped after changing version on the process
group to a newer version.

Thanks,

Bryan


On Tue, May 4, 2021 at 8:07 AM  wrote:
>
> Hello Team,
>
> We are using NiFi, NiFi registry and GIT integration. We are facing below 
> issues. Request you to please guide us on the below issues:
>
>
>   1.  Problem :  NiFi Registry create folder in GitLab with bucket name and 
> store all the flows inside the same bucket.
> Expectation: Create the hierarchical folder structure to store the flows.
>
> Example: NiFi flow 123 is stored inside pg ABC and ABC is child pg of XYZ. 
> This XYZ is versioned in NiFi Registry in bucket name ASD. Now in Git ASD 
> folder will have all information of XYZ,ABC,123 this is flat folder structure 
> hence this needs to be checked if it can create the following folder 
> structure: XYZ → ABC → 123
>
>
>
> Question: Is it possible to do this? create folder structure in GIT in 
> hierarchical form?
>
>
>
>
>
>
>   1.  In the NiFi deployment process using NiFi, NiFi registry for deploying 
> the flows from Dev to Test environment we need to have below things in place:
>
>
> a) If a processor is disabled in the dev environment then it must remain 
> disabled in the target cluster. (Current solution -  Via NiFi rest API check 
> the status of process groups)
> b) If a process group is deployed for the first time, the process group 
> should not be started automatically and remains stopped(Current solution -  
> Via NiFi rest API check the flow names of dev and test flows and know if the 
> flow is new flow or updated flow)
> c) A process group which is upgraded adopts the status from the target 
> cluster (stopped process group remains stopped, started process group should 
> be started) (Current solution -  Via NiFi rest API check the status of 
> process groups in target environment before the update and keep the same 
> after the update)
>
>
> Question: Any idea how can we do this apart from the above(bold highlighted) 
> mentioned solutions?
>
>
> Thanks & Regards,
> Shweta Soni | Consultant
> T-Systems ICT India Pvt. Ltd.
> A Deutsche Telekom Group Company.
> Address: 5th Floor, Panchshil Business Park,
> Tower-A, C.S. No:20, Balewadi High Street, Pune, Maharashtra, India 411045
> Mobile: +91 9767205291
> Email :  shweta.s...@t-systems.com
> Website: www.t-systems.com
>
> Life is  for Sharing
>

Re: Nifi authentication through Kerberos issues

[Bryan Bende]

> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at
> org.springframework.security.kerberos.authentication.sun.SunJaasKerberosClient.login(SunJaasKerberosClient.java:59)
> ... 82 common frames omitted
> Caused by: sun.security.krb5.internal.KrbApErrException: Message stream
> modified (41)
> at sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:101)
> at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
> at sun.security.krb5.KrbAsRep.decryptUsingPassword(KrbAsRep.java:139)
> at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310)
> at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:770)
> ... 95 common frames omitted
>
> On Wed, Mar 31, 2021 at 4:44 PM Derek Richardson  wrote:
>
> > Correct.
> >
> > # kinit admin@MY.REALM
> > Password for admin@MY.REALM:
> >
> > # klist
> > Ticket cache: FILE:/tmp/krb5cc_0
> > Default principal: admin@MY.REALM
> >
> > Valid starting   Expires  Service principal
> > 03/31/2021 22:42:10  04/01/2021 22:42:10  krbtgt/MY.REALM@MY.REALM
> >
> > On Wed, Mar 31, 2021, 1:13 PM Bryan Bende  wrote:
> >
> >> So from a terminal on the nifi server, you can run "kinit
> >> admin@MY.REALM" and enter the password and it works, and this same
> >> principal and password entered into NiFi's login screen does not work?
> >>
> >> On Wed, Mar 31, 2021 at 2:19 PM Derek Richardson 
> >> wrote:
> >> >
> >> > I'm working on transitioning a nifi instance we deploy with Kerberos and
> >> > I'm having some trouble authenticating. Everything looks correct, but
> >> when
> >> > I try to log in with any of my created users, I get an error message:
> >> >
> >> > The supplied username and password are not valid.
> >> >
> >> > Everything on nifi without https was working, and everything I've
> >> created
> >> > on the Kerberos side looks and works as expected, I just haven't been
> >> able
> >> > to get a user to log in to the Nifi UI.
> >> >
> >> > Here are some of my config files, is there anything I'm missing or have
> >> > incorrect?
> >> >
> >> > ---
> >> >
> >> > Authorizers.xml:
> >> > 
> >> > 
> >> > 
> >> > file-user-group-provider
> >> >
> >>  org.apache.nifi.authorization.FileUserGroupProvider
> >> > ./conf/users.xml
> >> > 
> >> >
> >> > 
> >> > 
> >> >
> >> > 
> >> > file-access-policy-provider
> >> >
> >> > org.apache.nifi.authorization.FileAccessPolicyProvider
> >> > file-user-group-provider
> >> > ./conf/authorizations.xml
> >> > admin@MY.REALM
> >> 
> >> > 
> >> > 
> >> > 
> >> > 
> >> >
> >> > 
> >> > managed-authorizer
> >> >
> >> > org.apache.nifi.authorization.StandardManagedAuthorizer
> >> > file-access-policy-provider
> >> > 
> >> >
> >> > 
> >> > file-provider
> >> > org.apache.nifi.authorization.FileAuthorizer
> >> > ./conf/authorizations.xml
> >> > ./conf/users.xml
> >> > admin@MY.REALM
> >> 
> >> > 
> >> >
> >> > 
> >> > 
> >> > 
> >> >
> >> > -
> >> >
> >> > Relevant nifi.properties:
> >> > nifi.security.user.authorizer=file-provider
> >> > nifi.security.user.login.identity.provider=kerberos-provider
> >> > # kerberos #
> >> > nifi.kerberos.krb5.file= /etc/krb5.conf
> >> > nifi.kerberos.service.principal=admin@MY.REALM
> >> > nifi.kerberos.service.keytab.location=/etc/kadm5.keytab
> >> >
> >> > -
> >> >
> >> > Login-identity-provider.xml
> >> > 
> >> > 
> >> > kerberos-provider
> >> > org.apache.nifi.kerberos.KerberosProvider
> >> > MY.REALM
> >> > 12 hours
> >> > 
> >> > 
> >> >
> >> > ---
> >> >
> >> > /etc/krb5.conf:
> >> > [logging]
> >> >  default = FILE:/var/log/krb5libs.log
> >> >  kdc = FILE:/var/log/krb5kdc.log
> >> >  admin_server = FILE:/var/log/kadmind.log
> >> >
> >> > [libdefaults]
> >> >  ticket_lifetime = 24h
> >> >  renew_lifetime = 7d
> >> >  forwardable = true
> >> >  default_realm = MY.REALM
> >> >
> >> > [realms]
> >> >  RO.INTERNAL = {
> >> >   kdc = nifi-djr5.ro.internal:88
> >> >   admin_server = nifi-djr5.my.realm:749
> >> >   default_domain = my.realm
> >> >  }
> >> >
> >> > [domain_realm]
> >> >  .my.realm = MY.REALM
> >> >  my.realm = MY.REALM
> >> >
> >> > [kdc]
> >> >  profile = /var/kerberos/krb5kdc/kdc.conf
> >> >
> >> > ---
> >> >
> >> > Any help would be greatly appreciated!
> >>
> >

Re: Nifi authentication through Kerberos issues

[Bryan Bende]

So from a terminal on the nifi server, you can run "kinit
admin@MY.REALM" and enter the password and it works, and this same
principal and password entered into NiFi's login screen does not work?

On Wed, Mar 31, 2021 at 2:19 PM Derek Richardson  wrote:
>
> I'm working on transitioning a nifi instance we deploy with Kerberos and
> I'm having some trouble authenticating. Everything looks correct, but when
> I try to log in with any of my created users, I get an error message:
>
> The supplied username and password are not valid.
>
> Everything on nifi without https was working, and everything I've created
> on the Kerberos side looks and works as expected, I just haven't been able
> to get a user to log in to the Nifi UI.
>
> Here are some of my config files, is there anything I'm missing or have
> incorrect?
>
> ---
>
> Authorizers.xml:
> 
> 
> 
> file-user-group-provider
> org.apache.nifi.authorization.FileUserGroupProvider
> ./conf/users.xml
> 
>
> 
> 
>
> 
> file-access-policy-provider
>
> org.apache.nifi.authorization.FileAccessPolicyProvider
> file-user-group-provider
> ./conf/authorizations.xml
> admin@MY.REALM
> 
> 
> 
> 
>
> 
> managed-authorizer
>
> org.apache.nifi.authorization.StandardManagedAuthorizer
> file-access-policy-provider
> 
>
> 
> file-provider
> org.apache.nifi.authorization.FileAuthorizer
> ./conf/authorizations.xml
> ./conf/users.xml
> admin@MY.REALM
> 
>
> 
> 
> 
>
> -
>
> Relevant nifi.properties:
> nifi.security.user.authorizer=file-provider
> nifi.security.user.login.identity.provider=kerberos-provider
> # kerberos #
> nifi.kerberos.krb5.file= /etc/krb5.conf
> nifi.kerberos.service.principal=admin@MY.REALM
> nifi.kerberos.service.keytab.location=/etc/kadm5.keytab
>
> -
>
> Login-identity-provider.xml
> 
> 
> kerberos-provider
> org.apache.nifi.kerberos.KerberosProvider
> MY.REALM
> 12 hours
> 
> 
>
> ---
>
> /etc/krb5.conf:
> [logging]
>  default = FILE:/var/log/krb5libs.log
>  kdc = FILE:/var/log/krb5kdc.log
>  admin_server = FILE:/var/log/kadmind.log
>
> [libdefaults]
>  ticket_lifetime = 24h
>  renew_lifetime = 7d
>  forwardable = true
>  default_realm = MY.REALM
>
> [realms]
>  RO.INTERNAL = {
>   kdc = nifi-djr5.ro.internal:88
>   admin_server = nifi-djr5.my.realm:749
>   default_domain = my.realm
>  }
>
> [domain_realm]
>  .my.realm = MY.REALM
>  my.realm = MY.REALM
>
> [kdc]
>  profile = /var/kerberos/krb5kdc/kdc.conf
>
> ---
>
> Any help would be greatly appreciated!

Re: [VOTE] Release Apache NiFi 1.13.1

[Bryan Bende]

+1 (binding)

- Verified everything in the standard release helper
- Setup secure standalone instance with SAML authentication

On Fri, Mar 12, 2021 at 10:17 AM Marton Szasz  wrote:
>
> +1 (non-binding)
>
> Followed the release helper guide, tested the binary with a simple flow.
>
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home: /usr/share/maven-bin-3.6
> Java version: 1.8.0_252, vendor: IcedTea, runtime: /opt/icedtea-bin-3.16.0/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "5.10.12-gentoo-x86_64", arch: "amd64",
> family: "unix"
>
>
> On Fri, 12 Mar 2021 at 13:35, Otto Fowler  wrote:
> >
> > +1
> >
> > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> > Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
> > Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
> > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
> >
> >
> > > On Mar 11, 2021, at 11:29, Joe Witt  wrote:
> > >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache NiFi
> > > 1.13.1.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1179
> > >
> > > The source being voted upon and the convenience binaries can be found at:
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/
> > >
> > > A helpful reminder on how the release candidate verification process 
> > > works:
> > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-1.13.1-RC1
> > > The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01
> > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01
> > >
> > > Checksums of nifi-1.13.1-source-release.zip:
> > > SHA256: 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac
> > > SHA512:
> > > 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/joewitt.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 48 issues were closed/resolved for this release:
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> > >
> > > Release note highlights can be found here:
> > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test. Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.13.1
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >

Re: Preconfiguring dynamic properties

[Bryan Bende]

I don't think it was the intent to pre-add a dynamic property. You
should be able to set a default value though, the user still has to
click the + icon to add the property though.

On Wed, Feb 24, 2021 at 12:02 PM Russell Bateman  wrote:
>
> I have a dynamic property in a custom processor that my down-streamers
> struggle a little bit to configure (requires newlines and a peculiar
> format). I would like to "preconfigure" a dynamic property as an example
> that they can either modify or erase to add their own. Most of them
> would probably just use what I preconfigure.
>
> The point is that I don't wish it to be a full-raging, static property.
> I want to see a trash can to the right of it. The trash can is not a
> feature of static properties.
>
> Is this possible, wrong-headed, what?
>
> Thanks,
> Russ

Re: Username/Password authentication

[Bryan Bende]

The available authentication mechanisms are...

- Client certificate
- Kerberos SPNEGO
- Keberos login identity provider
- LDAP login identity provider
- OpenID Connect
- Apache Knox SSO
- SAML (soon to be released)

You can also implement your own LoginIdentityProvider.

We currently don't provide one that stores usernames/passwords in NiFi.

On Fri, Feb 12, 2021 at 9:03 AM Edward Armes  wrote:
>
> Hi Sumant,
>
> The best way to do this would be using an approach known as Basic
> Authentication. information you need to enable this can be found in
> Administration guide which can be found here:
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
>
> Regards
>
> Edward
>
>
> On Fri, Feb 12, 2021 at 1:32 PM Sumanta Mishra
>  wrote:
>
> > Adding us...@nifi.apache.org
> >
> > On Fri, Feb 12, 2021 at 5:40 PM Sumanta Mishra <
> > sumanta.mis...@magicedtech.com> wrote:
> >
> > > Hi,
> > >
> > > Is there any direct and simple way to set username and password within
> > > Nifi? So that:
> > >
> > >- I can create users
> > >- Share the credentials with the users
> > >- Application should prompt authentication popup if they open the
> > >application
> > >- Login and use the application using their credentials
> > >
> > >
> > > LDAP authentication is bit difficult in ubuntu machine. I am using Azure
> > > Virtual Machine to user Nifi.
> > >
> > > Any suggestions?
> > >
> > > Regards,
> > > Sumant
> > >
> >

Re: [discuss] we need to enable secure by default...

[Bryan Bende]

open
> > > option'.
> > >
> > > Thanks
> > >
> > > On Wed, Feb 10, 2021 at 6:58 AM David Handermann <
> > exceptionfact...@gmail.com>
> > > wrote:
> > >
> > >> Having NiFi enforce authentication by default seems like the right way
> > to
> > >> go, especially given the capabilities of the system.
> > >>
> > >> Bryan raises a good point about storage of account information, so
> > weighing
> > >> the positives and negatives of various identity providers should be
> > >> considered.
> > >>
> > >> Following on Joe's point about disabling plain HTTP, one option could be
> > >> generating both client and server certificates and using Mutual TLS for
> > >> authentication.  This would obviously require installing the client
> > >> certificate in a browser, which is not trivial, but could be part of an
> > >> installation guide.  This approach definitely provides a high barrier of
> > >> entry of new users, but provides a strong level of security while
> > avoiding
> > >> the need for some other identity provider service to be configured.  As
> > >> others have mentioned, this seems necessary to address the situation of
> > >> someone installing NiFi without a full understanding of the
> > configuration
> > >> required, so it is important to keep the audience in mind when
> > evaluating a
> > >> solution.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> On Wed, Feb 10, 2021 at 7:39 AM Bryan Bende  wrote:
> > >>
> > >>> Just to clarify, I was not suggesting that we make a default secure
> > >>> setup that requires LDAP.
> > >>>
> > >>> I was just saying that in order to provide a default secure setup,
> > >>> we'd have to provide a login identity provider implementation that is
> > >>> backed by a file or database table, which in the past we decided
> > >>> against.
> > >>>
> > >>> On Wed, Feb 10, 2021 at 8:28 AM Russell Bateman  > >
> > >>> wrote:
> > >>>>
> > >>>> I second the concerns expressed, but second especially Bryan's
> > pointing
> > >>>> out that requiring LDAP/AD to be set up in order even to begin to use
> > >>>> our framework would be a bit onerous for developers just interested in
> > >>>> getting work done and a barrier to considering the framework should it
> > >>>> be erected a little too high. Should we at least glance at how this is
> > >>>> solved by the likes of other projects, Kafka and Cassandra come to
> > >> mind,
> > >>>> even if it means resorting to a store of a name or two? I didn't find
> > >>>> getting into developing with them a pain, but making me jump through
> > >> the
> > >>>> hoop of setting up LDAP may very well have changed that.
> > >>>>
> > >>>> These unsecure instances of NiFi out there are not our community's
> > >>>> fault. I suppose we're worried about getting splattered by bad press?
> > >>>>
> > >>>> On 2/10/21 5:47 AM, Bryan Bende wrote:
> > >>>>> I agree with the overall idea, although I would think it requires a
> > >>>>> major release to make this kind of change to the default behavior.
> > >>>>>
> > >>>>> Also, we have always avoided NiFi being a store of usernames and
> > >>>>> passwords, so we don't have a login provider that uses a local file
> > >> or
> > >>>>> a database, we've always said you connect to LDAP/AD for that.
> > >>>>>
> > >>>>> Obviously it can be implemented, but just pointing out that we'd have
> > >>>>> to change our stance here if we want to provide a default username
> > >> and
> > >>>>> password to authenticate with.
> > >>>>>
> > >>>>> On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande 
> > >>> wrote:
> > >>>>>> Mysql has been generating an admin password on default installs for,
> > >>> like,
> > >>>>>> forever. This workflow should be familiar for many users.
> > >>>>>>
> > >>>>>> I'd suggest taking the automation tooling into account and how a
> > >>> production
> > >>>>>> rollout (user-provided password) would fit into the workflow.
> > >>>>>>
> > >>>>>> Andrew
> > >>>>>>
> > >>>>>> On Tue, Feb 9, 2021, 8:15 PM Tony Kurc  wrote:
> > >>>>>>
> > >>>>>>> Joe,
> > >>>>>>> In addition to your suggestions, were you thinking of making this
> > >>> processor
> > >>>>>>> disabled by default as well?
> > >>>>>>>
> > >>>>>>> Tony
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, Feb 9, 2021, 11:04 PM Joe Witt  wrote:
> > >>>>>>>
> > >>>>>>>> Team
> > >>>>>>>>
> > >>>>>>>> While secure by default may not be practical perhaps ‘not
> > >> blatantly
> > >>> wide
> > >>>>>>>> open’ by default should be adopted.
> > >>>>>>>>
> > >>>>>>>> I think we should consider killing support for http entirely and
> > >>> support
> > >>>>>>>> only https.  We should consider auto generating a user and
> > >> password
> > >>> and
> > >>>>>>>> possibly server cert if nothing is configured and log the
> > >> generated
> > >>> user
> > >>>>>>>> and password.   Sure it could still be configured to be non secure
> > >>> but
> > >>>>>>> that
> > >>>>>>>> would truly be an admins fault.  Now its just ‘on’
> > >>>>>>>>
> > >>>>>>>> This tweet is a great example of why
> > >>>>>>>>
> > >>>>>>>> https://twitter.com/_escctrl_/status/1359280656174510081?s=21
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Who agrees?  Who disagrees?   Please share ideas.
> > >>>>>>>>
> > >>>>>>>> Thanks
> > >>>>>>>>
> > >>>
> > >>
> >
> >

Re: [discuss] we need to enable secure by default...

[Bryan Bende]

Just to clarify, I was not suggesting that we make a default secure
setup that requires LDAP.

I was just saying that in order to provide a default secure setup,
we'd have to provide a login identity provider implementation that is
backed by a file or database table, which in the past we decided
against.

On Wed, Feb 10, 2021 at 8:28 AM Russell Bateman  wrote:
>
> I second the concerns expressed, but second especially Bryan's pointing
> out that requiring LDAP/AD to be set up in order even to begin to use
> our framework would be a bit onerous for developers just interested in
> getting work done and a barrier to considering the framework should it
> be erected a little too high. Should we at least glance at how this is
> solved by the likes of other projects, Kafka and Cassandra come to mind,
> even if it means resorting to a store of a name or two? I didn't find
> getting into developing with them a pain, but making me jump through the
> hoop of setting up LDAP may very well have changed that.
>
> These unsecure instances of NiFi out there are not our community's
> fault. I suppose we're worried about getting splattered by bad press?
>
> On 2/10/21 5:47 AM, Bryan Bende wrote:
> > I agree with the overall idea, although I would think it requires a
> > major release to make this kind of change to the default behavior.
> >
> > Also, we have always avoided NiFi being a store of usernames and
> > passwords, so we don't have a login provider that uses a local file or
> > a database, we've always said you connect to LDAP/AD for that.
> >
> > Obviously it can be implemented, but just pointing out that we'd have
> > to change our stance here if we want to provide a default username and
> > password to authenticate with.
> >
> > On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande  wrote:
> >> Mysql has been generating an admin password on default installs for, like,
> >> forever. This workflow should be familiar for many users.
> >>
> >> I'd suggest taking the automation tooling into account and how a production
> >> rollout (user-provided password) would fit into the workflow.
> >>
> >> Andrew
> >>
> >> On Tue, Feb 9, 2021, 8:15 PM Tony Kurc  wrote:
> >>
> >>> Joe,
> >>> In addition to your suggestions, were you thinking of making this 
> >>> processor
> >>> disabled by default as well?
> >>>
> >>> Tony
> >>>
> >>>
> >>> On Tue, Feb 9, 2021, 11:04 PM Joe Witt  wrote:
> >>>
> >>>> Team
> >>>>
> >>>> While secure by default may not be practical perhaps ‘not blatantly wide
> >>>> open’ by default should be adopted.
> >>>>
> >>>> I think we should consider killing support for http entirely and support
> >>>> only https.  We should consider auto generating a user and password and
> >>>> possibly server cert if nothing is configured and log the generated user
> >>>> and password.   Sure it could still be configured to be non secure but
> >>> that
> >>>> would truly be an admins fault.  Now its just ‘on’
> >>>>
> >>>> This tweet is a great example of why
> >>>>
> >>>> https://twitter.com/_escctrl_/status/1359280656174510081?s=21
> >>>>
> >>>>
> >>>> Who agrees?  Who disagrees?   Please share ideas.
> >>>>
> >>>> Thanks
> >>>>

Re: [discuss] we need to enable secure by default...

[Bryan Bende]

I agree with the overall idea, although I would think it requires a
major release to make this kind of change to the default behavior.

Also, we have always avoided NiFi being a store of usernames and
passwords, so we don't have a login provider that uses a local file or
a database, we've always said you connect to LDAP/AD for that.

Obviously it can be implemented, but just pointing out that we'd have
to change our stance here if we want to provide a default username and
password to authenticate with.

On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande  wrote:
>
> Mysql has been generating an admin password on default installs for, like,
> forever. This workflow should be familiar for many users.
>
> I'd suggest taking the automation tooling into account and how a production
> rollout (user-provided password) would fit into the workflow.
>
> Andrew
>
> On Tue, Feb 9, 2021, 8:15 PM Tony Kurc  wrote:
>
> > Joe,
> > In addition to your suggestions, were you thinking of making this processor
> > disabled by default as well?
> >
> > Tony
> >
> >
> > On Tue, Feb 9, 2021, 11:04 PM Joe Witt  wrote:
> >
> > > Team
> > >
> > > While secure by default may not be practical perhaps ‘not blatantly wide
> > > open’ by default should be adopted.
> > >
> > > I think we should consider killing support for http entirely and support
> > > only https.  We should consider auto generating a user and password and
> > > possibly server cert if nothing is configured and log the generated user
> > > and password.   Sure it could still be configured to be non secure but
> > that
> > > would truly be an admins fault.  Now its just ‘on’
> > >
> > > This tweet is a great example of why
> > >
> > > https://twitter.com/_escctrl_/status/1359280656174510081?s=21
> > >
> > >
> > > Who agrees?  Who disagrees?   Please share ideas.
> > >
> > > Thanks
> > >
> >

Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)

[Bryan Bende]

+1 (binding)

- Verified everything in the release helper
- Tested enabling SAML authentication

Thanks Joe!

On Wed, Feb 3, 2021 at 1:10 PM karthick rn  wrote:
>
> +1 (non-binding)
>
> Verified the signature & checksum
> Verified the build works on CentOS 8.2 (Azure VM), OpenJDK 11.0.9.1, Maven
> 3.6.3
> Saw the copyright is already take care
>
> Thanks,
> Karthick
>
>
> On Wed, 3 Feb 2021 at 16:52, Robert Fellows  wrote:
>
> > +1 (non-binding)
> > Followed the release helper
> > Tested basic functionality
> > Apache Maven 3.6.3
> > Java version: 11.0.8, vendor: AdoptOpenJDK
> > OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
> >
> > Thanks for RM'ing, Joe.
> >
> > -- Rob
> >
> > On Tue, Feb 2, 2021 at 9:46 PM M Tien  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Went through the release helper
> > > Verified a successful build on
> > > Zulu OpenJDK 1.8.0_282
> > > Zulu OpenJDK 11.0.10
> > > Maven 3.6.2
> > > OS name: "mac os x", version: "10.15.4", arch: "x86_64", family: "mac"
> > > Ran a secure instance with OIDC using Google and Okta
> > > Verified a successful user logout and the JWT is invalidated (NIFI-7584)
> > > Tested with a flow using the CompressContent processor and verified
> > > dependent properties (NIFI-1121)
> > >
> > > Thanks,
> > > Margot
> > >
> > > > On Feb 2, 2021, at 6:23 PM, Chad Zobrisky  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Verified signatures, checksums, README, NOTICE and LICENSE
> > > >
> > > > Built with maven 3.6.3
> > > > Ubuntu 20.04.1
> > > > openjdk 11.0.9.1 and openjdk 1.8.0_275
> > > > Ran on a secure cluster with java 11.0.9.1 - all previous flows ran as
> > > > expected.
> > > >
> > > > Thanks,
> > > > Chad
> > > >
> > > >
> > > > On Tue, Feb 2, 2021 at 8:15 PM Mark Bean 
> > wrote:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> Verified signature and checksums
> > > >> Downloaded source and built with Java 8 and Java 11
> > > >>  mvn -T 1C clean install -Pcontrib-check,include-grpc
> > > >>  MacOS, 10.15.7
> > > >>  Java 11.0.9, x86_64
> > > >>  Java 1.8.0_282, x86_64
> > > >>  Maven 3.6.3
> > > >> Verified README, NOTICE and LICENSE (noting 2020 date in License)
> > > >> Ran in both Java 8 and Java 11 JVM with a _very_ basic flow. All basic
> > > >> functions and UI interaction appear normal.
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Feb 2, 2021 at 7:41 PM Muazma Zahid  wrote:
> > > >>
> > > >>>  +1 (non-binding)
> > > >>>
> > > >>> - Ran build with OpenJDK 1.8.0_275 on Linux
> > > >>> - Deployed cluster on Azure and tested standard flows with Blob,
> > ADLS,
> > > >> and
> > > >>> CosmosDB processors.
> > > >>>
> > > >>> Looks good to me.
> > > >>>
> > > >>> Thanks
> > > >>> Muazma
> > > >>>
> > > >>> On Tue, Feb 2, 2021 at 4:35 PM Nathan Gough 
> > > wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  - Verified signature and checksums again
> > >  - Ran build with
> > >  java -version
> > > 
> > >  openjdk version "1.8.0_282"
> > > 
> > >  OpenJDK Runtime Environment (Zulu 8.52.0.23-CA-macosx) (build
> > >  1.8.0_282-b08)
> > > 
> > >  OpenJDK 64-Bit Server VM (Zulu 8.52.0.23-CA-macosx) (build
> > 25.282-b08,
> > >  mixed mode)
> > > 
> > >  - Tested secure clusters with TLS ZK, secure Site-to-Site,
> > InvokeHTTP,
> > >  ListenHTTP
> > > 
> > >  On Tue, Feb 2, 2021 at 7:20 PM Joey Frazee  > >  .invalid>
> > >  wrote:
> > > 
> > > > +1 (non-binding)
> > > >
> > > > - Verified checksums, signatures, and commit id
> > > > - Ran builds with Java 1.8 and 11 on Linux and macOS, and validated
> > > >> RPM
> > > > build profile
> > > > - Tested cluster coordination and state management with both
> > embedded
> > > >>> and
> > > > external ZooKeepers with TLS enabled and disabled
> > > > - Verified fix for PutAzureBlobStorage OOME and tested Blob and
> > Queue
> > > > storage with Azurite emulator
> > > >
> > > > -joey
> > > >
> > > >> On Feb 2, 2021, at 3:16 PM, Sushil Kumar 
> > > >> wrote:
> > > >>
> > > >> +1 (non-binding) Release this package as nifi-1.13.0
> > > >>
> > > >> Deployed this via helm chart(
> > > >> https://github.com/sushilkm/nifi-chart)
> > >  on
> > > >> kubernetes.
> > > >> Thank you to all the contributors and reviewers.
> > > >>
> > > >>
> > > >>> On Tue, Feb 2, 2021 at 11:02 AM Matt Burgess <
> > > >> mattyb...@apache.org>
> > > > wrote:
> > > >>>
> > > >>> +1 (binding) Release this package as nifi-1.13.0
> > > >>>
> > > >>> Ran through release helper and tried various flows using some of
> > > >> the
> > > >>> new features and capabilities added in 1.13.0.  Thanks for RM'ing
> > > >>> Joe!
> > > >>>
> > >  On Mon, Feb 1, 2021 at 8:10 PM Joe Witt 
> > > >>> wrote:
> > > 
> > >  Hello,
> > > 
> > >  I am pleased to be calling this vote for the source 

Re: java api for changing parameter context

[Bryan Bende]

You can open Chrome Dev Tools while using the UI and perform whichever
operations you are interested in on a parameter context and you can
see the requests made to the REST API.

If you are interested in REST API docs, there is a link on the left
side of the main docs page:

https://nifi.apache.org/docs.html

Parameter contexts are under Controller.

On Wed, Jan 27, 2021 at 12:08 PM u...@moosheimer.com  
wrote:
>
> Couldn't find any information about changing parameters via REST API.
> Do you have any example?
>
> Mit freundlichen Grüßen / best regards
> Kay-Uwe Moosheimer
>
> > Am 27.01.2021 um 15:17 schrieb Russell Bateman :
> >
> > Wait! Can't this be done using the ReST APIs?
> >
> >> On 1/27/21 3:24 AM, u...@moosheimer.com wrote:
> >> Hello NiFi-Core-Team,
> >>
> >> Are you planning to create a high-level Java API for setting (and
> >> clearing) individual parameters in the parameter context, so we can use
> >> this API in processor development?
> >>
> >> Example:
> >> setParameter(string contextName, string parameterName, string
> >> parameterValue, boolean sensitive);
> >> deleteParameter(string contextName, string parameterName);
> >>
> >> Some of our customers have systems with weekly changing parameter values
> >> and/or access passphrase.
> >> Apart from these nothing changes in the system and the changes can be
> >> automated with self written processor.
> >>
> >> Best regards,
> >> Kay-Uwe Moosheimer
> >>
> >>
> >>
> >
>

Re: java api for changing parameter context

[Bryan Bende]

Hello,

There are no plans to allow an extension (processor/cs/reporting task)
to change parameter contexts.

The REST API can be used to automate these tasks and there are already
commands in NiFi CLI to help make this easier...

nifi set-param
nifi delete-param
nifi list-param-contexts
nifi get-param-context
nifi create-param-context
nifi delete-param-context
nifi merge-param-context
nifi import-param-context
nifi pg-get-param-context
nifi pg-set-param-context

Thanks,

Bryan

On Wed, Jan 27, 2021 at 9:17 AM Russell Bateman  wrote:
>
> Wait! Can't this be done using the ReST APIs?
>
> On 1/27/21 3:24 AM, u...@moosheimer.com wrote:
> > Hello NiFi-Core-Team,
> >
> > Are you planning to create a high-level Java API for setting (and
> > clearing) individual parameters in the parameter context, so we can use
> > this API in processor development?
> >
> > Example:
> > setParameter(string contextName, string parameterName, string
> > parameterValue, boolean sensitive);
> > deleteParameter(string contextName, string parameterName);
> >
> > Some of our customers have systems with weekly changing parameter values
> > and/or access passphrase.
> > Apart from these nothing changes in the system and the changes can be
> > automated with self written processor.
> >
> > Best regards,
> > Kay-Uwe Moosheimer
> >
> >
> >
>

Minimum Maven Version

[Bryan Bende]

Since the beginning of the project our README has said "Maven >= 3.1",
which is obviously very old now. I was recently running into a problem
building main with Maven 3.3.9, that looks related to some new Maven
plugins in main depending on Maven 3.5.x apis. Switching to Maven
3.5.x resolved the issue, and I also regularly build with 3.6.3
without any issues.

I'd like to suggest that we state that the minimum Maven version is >=
3.5 now. This really doesn't mean too much other than updating the
README and setting the expectation of what should work.

If you are interested in when different versions of Maven have been
released, this page provides a nice listing -
https://maven.apache.org/docs/history.html

Thanks,

Bryan

Re: Extending StandardOidcIdentityProvider

[Bryan Bende]

Hello,

1) I don't think NiFi currently supports the client id + x509
scenario, is this part of the OIDC standard? If so then maybe it can
be an improvement that is implemented.

2) The OIDC code in NiFi is not part of an extension point, so you
can't just plug in your own version. You would have to modify the code
in NiFi and rebuild the nifi-framework-nar with your changes.

-Bryan

On Fri, Jan 15, 2021 at 1:25 PM Vijay Jammi  wrote:
>
> Hello there,
>
> I am trying to enable OIDC [OpenIDConnect/OAuth2.0] for our on prem Nifi
> with our on prem Identity Provider [Microsoft ADFS].
>
> Now, it looks like Nifi's authorization code flow requires a client id [
> nifi.security.user.oidc.client.id] and client secret
> [nifi.security.user.oidc.client.secret] to be able to exchange
> Authorization Code for an Access and Id Token. However, our Authorization
> Server only supports client id and x509 client certificate based
> authentication [Client Assertion] for the exchange. So my question here is
>
>  1. Is there way to configure Nifi for client id and x509 client
> certificate for the exchange?
>  2. If not, how can we extend Nifi for our need?
>
> I am new to Nifi so please excuse me if this is trivial within the Nifi
> development. I see a StandardOidcIdentityProvider under nifi-web-security.
> Can I override the default functionality by making a custom bundle to
> override or will I need to rebuild the bundle associated to
> nifi-web-security and drop it into the Nifi lib?  Any guidance will be much
> appreciated.
>
> Thank you in advance.
>
> Vijay Jammi

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Bryan Bende]

Yea usually when you get RAT issues and the files referenced are in
target and other project related config files, it is because that
module was actually deleted and those files weren't under source
control so when you pulled the latest code it kept a copy of them
locally.


On Thu, Jan 14, 2021 at 8:48 AM Mark Payne  wrote:
>
> The nifi-docker/dockermaven-stateless module no longer exists. So you’d need 
> to do a “git clean -fd” I think is the command to remove directories that no 
> longer exist on your branch.
>
> > On Jan 14, 2021, at 8:44 AM, Mark Bean  wrote:
> >
> > The grpc-bundle is normally excluded from builds unless activated by a
> > profile. So, it is often left excluded from 'mvn clean'. You can clean it
> > up manually, or by
> >
> > mvn -Pinclude-grpc clean
> >
> > That will take care of many of your issues. As for the ones in nifi-docker,
> > you may have to go in there and clean that one individually.
> >
> > -Mark
> >
> >
> > On Thu, Jan 14, 2021 at 4:30 AM Chris Sampson
> >  wrote:
> >
> >> Running the contrib-check against current main fails due to "too many files
> >> with unapproved licences".
> >>
> >> Command I'm running: mvn -T 2.0C -Pcontrib-check clean install
> >>
> >> mvn --version:
> >>
> >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> >> Java version: 11.0.9, vendor: AdoptOpenJDK, runtime:
> >> .../java/11.0.9.hs-adpt
> >> Default locale: en_GB, platform encoding: UTF-8
> >> OS name: "linux", version: "5.8.0-36-generic", arch: "amd64", family:
> >> "unix"
> >>
> >>
> >> RAT reports failing:
> >>
> >>   - nifi-docker
> >>
> >> nifi-docker/dockermaven-stateless/target/maven-archiver/pom.properties
> >> nifi-docker/dockermaven-stateless/target/.plxarc
> >> nifi-docker/dockermaven-stateless/dockermaven-stateless.iml
> >> nifi-docker/dockermaven-stateless/.settings/org.eclipse.m2e.core.prefs
> >> nifi-docker/dockermaven-stateless/.project
> >>
> >>   - nifi-nar-bundles
> >>
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/.settings/org.eclipse.core.resources.prefs
> >> nifi-nar-bundles/nifi-grpc-bundle/.settings/org.eclipse.m2e.core.prefs
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-nar/.settings/org.eclipse.m2e.core.prefs
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-nar/.project
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/.project
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/field_mask.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/api.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/struct.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/duration.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/source_context.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/wrappers.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/timestamp.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/empty.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/compiler/plugin.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/type.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/any.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/target/protoc-dependencies/f83446e3c1ecbf8612db34fa4a039ef0/google/protobuf/descriptor.proto
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/.settings/org.eclipse.core.resources.prefs
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/.settings/org.eclipse.jdt.core.prefs
> >>
> >> nifi-nar-bundles/nifi-grpc-bundle/nifi-grpc-processors/.settings/org.eclipse.m2e.core.prefs
> >> nifi-nar-bundles/nifi-grpc-bundle/.project
> >>
> >>
> >> Would appear someone's accidentally pushed some Eclipse settings/project
> >> files and some protobuf files (generated in the build?) aren't being
> >> ignored.
> >>
> >>
> >> ---
> >> *Chris Sampson*
> >> IT Consultant
> >> chris.samp...@naimuri.com
> >> 
> >>
> >>
> >> On Wed, 13 Jan 2021 at 18:00, Joe Witt  wrote:
> >>
> >>> Otto/Chris - i see your notes and will see if I can 

Re: NiFi Registry policies

[Bryan Bende]

This is expected behavior...

Saving or pulling the flow is happening on behalf of the end user in
nifi which is being proxied by the nifi server over to registry. In
this case, the nifi server only needs proxy permissions.

Checking the status of all the versioned PGs happens in the background
as the nifi server itself, so it needs read access to all buckets.

On Wed, Jan 6, 2021 at 9:22 AM Mark Bean  wrote:
>
> I created a new bucket in NiFi Registry without making it publicly visible.
> Then, I added myself to the policies on the bucket with Read, Write and
> Delete permissions; I did not add the Apache NiFi server to the policies.
>
> In Apache NiFi, I added a flow to the Registry. I could also pull this flow
> from the Registry duplicating the process group in the flow.
>
> The status of the version controlled process group indicates "?" (sync
> failure). In the nifi-registry-app.log, I see a corresponding INFO message
> indicating the Apache NiFi server "does not have permission to access the
> requested resource" where the resource is the bucket mentioned above. I
> expect this since I have not yet added the NiFi server to the policies for
> the bucket.
>
> Is this expected behavior?
>
> I would expect that the flow could not be created in the bucket nor pulled
> from the bucket until the NiFi server is added to the policies for the
> bucket. Yet, I was able to perform both operations.
>
> NiFi Registry, 0.8.0
> Apache NiFi 1.11.4
>
> Thanks,
> Mark

Re: Jira contributor access

[Bryan Bende]

Hello,

I have added you as a contributor in JIRA.

Thanks,

Bryan

On Wed, Dec 9, 2020 at 10:45 AM Marton Szasz  wrote:
>
> Hi,
>
> You can subscribe to the lists by sending an email to the subscribe
> address, dev-subscr...@nifi.apache.org in this case.
>
> I can't add you to Jira but I'm sure someone with the necessary rights will
> help you out soon.
>
> Marton
>
> On Wed, 9 Dec 2020 at 15:20, Yuming Hsieh  wrote:
>
> > Hi,
> >
> > I would like to subscribe to the dev mailing list and get access to Jira.
> >
> > Jira account: yuminghsiehus
> >
> > Thanks,
> > ,Yuming
> >

Re: Usage of NiFI CLI create-service

[Bryan Bende]

Hello,

We need to add documentation around this, but right now you can see
examples in the JIRA which added these commands:

https://issues.apache.org/jira/browse/NIFI-6112

Thanks,

Bryan

On Thu, Dec 3, 2020 at 10:31 AM Huan Yan  wrote:
>
> Hello,
>
> I am currently trying to make a controller service using the NiFi CLI with
> the create-service argument. I have already been able to do other CLI
> commands such as pull from the registry but I cannot figure out how to be
> able to define the controller service in the input file.
>
> It seems that the input file is supposed to be JSON but what do I need to
> put for the fields? For instance I am trying to make a DBCPConnectionPool
> with the CLI. How do I map these fields in a JSON file by name?
>
> Thanks for your help

Re: urgent Info required on configure Maximum Timer Driven Thread Count

[Bryan Bende]

Hello,

Please don't label things as "urgent". This is a place to get free
help, and if someone knows the answer to your question, then they will
respond whenever they have time.

Thanks,

Bryan

On Wed, Dec 2, 2020 at 4:11 AM Chris Sampson
 wrote:
>
> To change the setting, if that's what you're after:
>
>- NiFi UI main "hamburger" menu (top-right of screen)
>- Controller Services
>- General
>- Maximum Timer Driven Thread Count
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
>
>
> On Wed, 2 Dec 2020 at 07:58, Peter Turcsanyi  wrote:
>
> > Hi Ganesh,
> >
> > The default setting is 10.
> > A typical setting is 2 x CPU cores on your machine.
> > It is used here in the code:
> >
> > https://github.com/apache/nifi/blob/c29cced269dcce28fb9ba034025d01e76a79b037/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L249
> >
> > I hope this helps. Not sure which one you are interested in.
> >
> > Best,
> > Peter Turcsanyi
> >
> > On Wed, Dec 2, 2020 at 7:51 AM Ganesh, B (Nokia - IN/Bangalore) <
> > b.gan...@nokia.com> wrote:
> >
> > > Hi ,
> > >
> > >
> > >
> > > The default of Maximum Timer Driven Thread Count  is set to 10 , I could
> > > not find the place or file which we are using that value 10 .
> > >
> > >
> > >
> > > Could ypu please help me to get that default configuration setting ?
> > >
> > >
> > >
> > > Thanks & Regards,
> > >
> > > Ganesh.B
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >

Re: Jira contributor access

[Bryan Bende]

Hello,

I have added you as a contributor, you should be able to assign
yourself tickets now.

Thanks,

Bryan

On Wed, Dec 2, 2020 at 8:21 AM David Handermann
 wrote:
>
> Hi Cory,
>
> Thanks for submitting the issue and requesting access to Jira, I'm sure one
> of the project maintainers will be able to add you soon.
>
> I added a comment to the issue you submitted and I would be interested to
> hear more details on your proposed implementation.  As mentioned in the
> comment, I have another open issue that is closely related to S/MIME
> handling, and includes some Controller Services that could be useful in
> implementing signed S/MIME emails.  Feel free to provide your comments on
> the related issue and PR in GitHub.
>
> Regards,
> David Handermann
>
> On Tue, Dec 1, 2020 at 10:00 PM cWix Software 
> wrote:
>
> > Hi there,
> >
> > I would like to work on Jira ticket NIFI-8059.
> >
> > I was reading you had to give me permissions so I could. My login is
> > cwixom.
> >
> > Please let me know if there's anything else I need to provide.
> >
> > Thanks!
> >
> > Cory Wixom
> >

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Bryan Bende]

Chris,

The illegal reflective access warnings are expected when running on Java 11.

The second warning about the provenance repo seems like a possible
issue due to some recent refactorings made for stateless, we should
take a look at that.

Thanks,

Bryan

On Tue, Dec 1, 2020 at 3:58 AM Chris Sampson
 wrote:
>
> Just spotted the following during startup of NiFi 1.13.0-SNAPSHOT on my
> local machine, which might be worth looking at (although I've not deleted
> and re-created the binaries in a little while, so I could have something
> wrong in my /lib folder or such):
>
> 2020-12-01 08:52:48,382 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> WARNING: An illegal reflective access operation has occurred
> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> (file:.../nifi-1.13.0-SNAPSHOT/lib/java11/jaxb-impl-2.3.0.jar) to method
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> WARNING: All illegal access operations will be denied in a future release
>
> Also:
> 2020-12-01 08:52:52,486 WARN [main]
> o.a.n.n.StandardExtensionDiscoveringManager Failed to register extension
> org.apache.nifi.provenance.VolatileProvenanceRepository due to: Attempt was
> made to load org.apache.nifi.provenance.VolatileProvenanceRepository from
> org.apache.nifi:nifi-provenance-repository-nar:1.13.0-SNAPSHOT but that
> class name is already loaded/registered from
> org.apache.nifi:nifi-stateless-nar:1.13.0-SNAPSHOT and multiple versions
> are not supported for this type
>
> These log messages appear just before the NAR extraction logs during
> instance startup.
>
> Don't know whether these are things that should be fixed before 1.13.0
> and/or have already been looked into, but figured I'd mention them so
> they're at least known about.
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
>
>
> On Mon, 30 Nov 2020 at 18:50, Nissim Shiman 
> wrote:
>
> >  Hello Nifi Team,
> >
> > NIFI-7738 [1]  https://github.com/apache/nifi/pull/4563
> > has already been reviewed/tested by a fellow contributer
> > I would greatly appreciate a committer moving this to main for 1.13.0
> > Thanks,
> > Nissim Shiman
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-7738
> > On Sunday, November 29, 2020, 07:12:12 PM EST, Matt Burgess <
> > mattyb...@apache.org> wrote:
> >
> >  I just merged Mike's Cassandra DMC PR, reviewing the graph stuff now.
> >
> > I have a bunch of open PRs [1], I don't think any are blockers for
> > 1.13.0 but thought I'd take this opportunity to find some reviewers :)
> >
> > Regards,
> > Matt
> >
> > [1] https://github.com/apache/nifi/pulls/mattyb149
> >
> > On Sat, Nov 28, 2020 at 10:13 AM Phillip Grenier 
> > wrote:
> > >
> > > I wouldn't mind https://github.com/apache/nifi/pull/4554 making it in.
> > Have
> > > seen a couple more instances where people have created custom processors
> > to
> > > work around this PR.
> > >
> > > On Thu, Nov 26, 2020 at 11:10 AM Mike Thomsen 
> > > wrote:
> > >
> > > > Also, for most of us (committers and community members), it's a
> > > > holiday season so might be good to delay a release to early January
> > > > anyway. Just a thought.
> > > >
> > > > On Thu, Nov 26, 2020 at 11:08 AM Mike Thomsen 
> > > > wrote:
> > > > >
> > > > > https://github.com/apache/nifi/pull/4638 - This is a big increase in
> > > > > graph functionality, and I want to give him a chance to get his first
> > > > > contribution in with 1.13 (we may have some more of his work we can
> > > > > open source).
> > > > >
> > > > > Matt's also reviewing some of my Cassandra and graph tickets as well.
> > > > > I wouldn't call those **blockers**, but having the Cassandra DMC
> > fully
> > > > > fleshed out would be nice.
> > > > >
> > > > > On Thu, Nov 26, 2020 at 9:10 AM Otto Fowler  > >
> > > > wrote:
> > > > > >
> > > > > >  NIFI-7795 https://github.com/apache/nifi/pull/4656
> > > > > > NIFI-7761 https://github.com/apache/nifi/pull/4513
> > > > > > NIFI-2072 https://github.com/apache/nifi/pull/4384
> > > > > >
> > > > > > From: Pierre Villard 
> > > > > > 
> > > > > > Reply: dev@nifi.apache.org  <
> > dev@nifi.apache.org>
> > > > > > Date: November 26, 2020 at 04:55:56
> > > > > > To: dev@nifi.apache.org   > >
> > > > > > Subject:  [DISCUSS] Release of Apache NiFi 1.13.0
> > > > > >
> > > > > > Hi community,
> > > > > >
> > > > > > Starting a discussion thread around the release 

Re: ListSftp/ListFtp Enhancements

[Bryan Bende]

Regarding adding support for expression language...

You could use parameter contexts instead of variables, all properties
can reference parameters by default so no need to update property
descriptors, just set to #{SearchRecursively} to reference the
parameter.

In my opinion this would be better than removing the use of
allowableValues on the Search Recursively just to support expression
language.

Parameters are introduced in nifi 1.10.0.

On Tue, Nov 17, 2020 at 11:32 PM Ganesh, B (Nokia - IN/Bangalore)
 wrote:
>
> We have below requirement to implement in apache Nifi , could you please let 
> me know your view .
>
>
>   1.  CNFI ListSFTP/ListFTP does not have an option to choose which files to 
> list first (older/newer)
> In NIFI, ListSFTP/ListFTP processors supporting an option to choose which 
> files to list first (older/newer) And also this new field should support NiFi 
> Expression Language.
> Use case is the following:
> We have a flow that is listing and fetching files from an external 
> environment that can have thousands of files to be collected. There is a need 
> to choose (depending on the environment) if we want to collect older files 
> first or newer files first.
> We have a process group that we instance with a few set of variables via API, 
> and one of them should be this Transfer New Files First (variable 
> TransferNewFilesFirst)
> With the proposed feature, the ListSFTP/ListFTP processors would have this 
> new field and inherit the variable e.g. ${TransferNewFilesFirst} and use it 
> for listing.
> Solution:
> Added new property descriptor - "Transfer New Files First" in the 
> ListSFTP/FTP processor. If true - then list the latest files first, else - 
> list the latest files at the end. And also supports expression language 
> (AbstractListProcessor.java) and added the same property in the ListSFTP.java 
> and ListFTP.java files.
> // new property descriptor
> public static final PropertyDescriptor TRANSFER_NEW_FILES_FIRST = new 
> PropertyDescriptor.Builder()
> .name("transfer-new-files-first")
> .displayName("Transfer New Files First")
> .description("If true, will list the newest resource first")
> .required(true)
> .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
> 
> .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
> .defaultValue("true")
> .build();
>
> // to list the files in ascending or descending order
> final boolean fileOrderController = 
> context.getProperty(TRANSFER_NEW_FILES_FIRST).evaluateAttributeExpressions().asBoolean();
>  for (Long timestamp : 
> (fileOrderController?orderedEntries.descendingKeySet():orderedEntries.keySet()))
>  {
> List entities = orderedEntries.get(timestamp);
> if 
> (timestamp.equals(lastProcessedLatestEntryTimestampMillis)) {
>// Filter out previously processed entities.
>entities = entities.stream().filter(entity -> 
> !latestIdentifiersProcessed.contains(entity.getIdentifier())).collect(Collectors.toList());
> }
>
> for (T entity : entities) {
> // Create the FlowFile for this path.
> final Map attributes = 
> createAttributes(entity, context);
> FlowFile flowFile = session.create();
> flowFile = session.putAllAttributes(flowFile, 
> attributes);
> session.transfer(flowFile, REL_SUCCESS);
> flowfilesCreated++;
> }
> }
> }
>
> 2. CNFI ListSFTP/ListFTP does not support Express Language for Search 
> Recursively field
> In NIFI, ListSFTP/ListFTP processors supporting the NiFi Expression Language 
> for the Search Recursively field
> Use case is the following:
> We have a process group that we instance with a few set of variables via API, 
> and one of them is the Search Recursively (variable SearchRecursively)
> With the proposed feature, the ListSFTP/ListFTP processors would inherit the 
> variable e.g. ${SearchRecursively} and use it for listing.
> Having the feature would allow to instance the process group with the 
> required variables from the get-go without further reconfigurations, thus 
> making it a bit more reusable.
> Solution:
> Added "BOOLEAN_VALIDATOR" and "ExpressionLanguageScope" in the "Search 
> Recursively" property. (FileTransfer.java, ListFile.java, FTPTransfer.java, 
> SFTPTransfer.java)
> public static final PropertyDescriptor RECURSIVE_SEARCH = new 
> PropertyDescriptor.Builder()
> .name("Search Recursively")
> .description("If true, will pull files from arbitrarily nested 
> subdirectories; otherwise, will not traverse subdirectories")
> .required(true)
> .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
> 
> 

Re: Secure parameters in invokeHttp header

[Bryan Bende]

Hello,

Since the dynamic properties of InvokeHttp are not marked as sensitive
properties, there currently isn't a way to add a sensitive header
value.

There is a JIRA to improve this experience -
https://issues.apache.org/jira/browse/NIFI-7310

Thanks,

Bryan

On Thu, Nov 5, 2020 at 11:02 AM Midhun Mohan  wrote:
>
> Hi all,
>
> I am looking for options on how to secure parameters passed in the header
> for this processor, looks like it currently has it as plain text in the
> processor. Whoever has access to the system can grab and use it.
>
> How can I restrict this?
>
> --
>
>
> Regards,
> Midhun Mohan

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

Sorry, the one list was meant to be...

a) system properties
b) environment variables
c) process-group variable registry
d) file-based variable registry

On Mon, Oct 19, 2020 at 4:35 PM Bryan Bende  wrote:
>
> I think there is some confusion based on terminology...
>
> A given property has an expression language scope defined, which can
> be "flow file attributes" or "variable registry only". This was
> created mostly for documentation purposes so that users could look at
> a property in the docs and see "expression language supported: true"
> and know which values they could reference. What it really comes down
> to in the code is the difference between
> "someProperty.evaluateAttributeExpressions()" and
> "someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
> reference a value from a flow file or not.
>
> In the case of "variable registry only", the value can come from any
> of the following...
> a) system properties
> b) expression language
> c) process-group variable registry
> d) file-based variable registry
>
> When people have stated that "variables are being deprecated in favor
> of parameters", they are referring to the last two items (c & d).
>
> The reason being that parameters solved several short-comings of those
> two options...
>
> - the ability to store sensitive values encrypted
> - the ability to reference them from any property using a new syntax
> #{...}, not long dependent on component developer saying
> "expressionLanguageSupported(true)" on the descriptor
> - the ability to create policies for which users/groups could
> reference a set of parameters
> - the hierarchical ambiguity of what "${foo}" actually resolves to
>
> If you just want to use environment variables, why use parameter
> contexts? Expression language has always offered access to env vars
> and still will going forward.
>
> The one  argument I can see is that not all properties support
> expression language, so using parameters gives you a way around that.
>
>
>
>
>
> On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
>  wrote:
> >
> > Being based primarily in Docker containers and having experience with both
> > Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> > environment variables or files) and Docker Swarm (which only handles
> > secrets as environment variables), I'd have definitely been wanting this
> > before moving from Variables to Parameters if I was still in Swarm (or
> > Docker Compose/straight up Docker).
> >
> > It's certainly possible to script creating/updating Parameters via the
> > Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> > possible as a Job in Kubernetes, for example). So environment variables
> > could save the day in that instance.
> >
> > I guess one likely problem (but no different to how I guess the Variable
> > Registry uses env vars) would be how NiFi will handle changes to the env
> > vars - does it:
> >
> >- ignore them until instance restart, which could lead to maintainer
> >confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
> >failing in NiFi)
> >- alert the maintainer to the fact that the env var has changed and a
> >Parameter needs updating, with the new value being used after all
> >associated processors/controllers have been restarted
> >- automatically attempt to update the parameters by restarting all
> >associated processors/controllers, which I'd guess would be a bit 
> > dangerous
> >for interrupting in-flow data, etc.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.samp...@naimuri.com
> > <https://www.naimuri.com/>
> >
> >
> > On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:
> >
> > > Access to environment variables directly from expression language is
> > > not being removed.
> > >
> > > The discussion is about whether a parameter value should be able to
> > > use expression language to reference an environment variable.
> > >
> > > For example, processor property has #{keystore.password} -> parameter
> > > "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> > > the password from an environment variable.
> > >
> > >
> > >
> > > On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
> > > wrote:
> > > >
> > > > Chad,
> > > >
> > > > So far I thought that

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

I think there is some confusion based on terminology...

A given property has an expression language scope defined, which can
be "flow file attributes" or "variable registry only". This was
created mostly for documentation purposes so that users could look at
a property in the docs and see "expression language supported: true"
and know which values they could reference. What it really comes down
to in the code is the difference between
"someProperty.evaluateAttributeExpressions()" and
"someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
reference a value from a flow file or not.

In the case of "variable registry only", the value can come from any
of the following...
a) system properties
b) expression language
c) process-group variable registry
d) file-based variable registry

When people have stated that "variables are being deprecated in favor
of parameters", they are referring to the last two items (c & d).

The reason being that parameters solved several short-comings of those
two options...

- the ability to store sensitive values encrypted
- the ability to reference them from any property using a new syntax
#{...}, not long dependent on component developer saying
"expressionLanguageSupported(true)" on the descriptor
- the ability to create policies for which users/groups could
reference a set of parameters
- the hierarchical ambiguity of what "${foo}" actually resolves to

If you just want to use environment variables, why use parameter
contexts? Expression language has always offered access to env vars
and still will going forward.

The one  argument I can see is that not all properties support
expression language, so using parameters gives you a way around that.





On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
 wrote:
>
> Being based primarily in Docker containers and having experience with both
> Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> environment variables or files) and Docker Swarm (which only handles
> secrets as environment variables), I'd have definitely been wanting this
> before moving from Variables to Parameters if I was still in Swarm (or
> Docker Compose/straight up Docker).
>
> It's certainly possible to script creating/updating Parameters via the
> Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> possible as a Job in Kubernetes, for example). So environment variables
> could save the day in that instance.
>
> I guess one likely problem (but no different to how I guess the Variable
> Registry uses env vars) would be how NiFi will handle changes to the env
> vars - does it:
>
>- ignore them until instance restart, which could lead to maintainer
>confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
>failing in NiFi)
>- alert the maintainer to the fact that the env var has changed and a
>Parameter needs updating, with the new value being used after all
>associated processors/controllers have been restarted
>- automatically attempt to update the parameters by restarting all
>associated processors/controllers, which I'd guess would be a bit dangerous
>for interrupting in-flow data, etc.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> <https://www.naimuri.com/>
>
>
> On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:
>
> > Access to environment variables directly from expression language is
> > not being removed.
> >
> > The discussion is about whether a parameter value should be able to
> > use expression language to reference an environment variable.
> >
> > For example, processor property has #{keystore.password} -> parameter
> > "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> > the password from an environment variable.
> >
> >
> >
> > On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
> > wrote:
> > >
> > > Chad,
> > >
> > > So far I thought that only the NiFi variables are deprecated and access
> > to environment variables will still be possible.
> > >
> > > If this is not the case, then I agree with you. It should definitely be
> > possible to access environment variables. Otherwise I can't imagine how to
> > refer to the hostname or the current JAVA path or ... or ... or on each
> > node?!
> > >
> > > Mit freundlichen Grüßen / best regards
> > > Kay-Uwe Moosheimer
> > >
> > > > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> > > >
> > > > Andy,
> > > >
> > > > Thanks for the response!
> > > >

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

Access to environment variables directly from expression language is
not being removed.

The discussion is about whether a parameter value should be able to
use expression language to reference an environment variable.

For example, processor property has #{keystore.password} -> parameter
"keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
the password from an environment variable.



On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com  wrote:
>
> Chad,
>
> So far I thought that only the NiFi variables are deprecated and access to 
> environment variables will still be possible.
>
> If this is not the case, then I agree with you. It should definitely be 
> possible to access environment variables. Otherwise I can't imagine how to 
> refer to the hostname or the current JAVA path or ... or ... or on each node?!
>
> Mit freundlichen Grüßen / best regards
> Kay-Uwe Moosheimer
>
> > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> >
> > Andy,
> >
> > Thanks for the response!
> >
> > When I was thinking through this the deprecation of variables was
> > definitely on my mind but the fact that it already had direct access to
> > environment variables was the simplest path. I think it does make more
> > sense to add access to environment variables to the parameter context, or
> > allowing a specific scope just for environment variables in the
> > expression language.
> >
> > I think giving access to environment variables actually allows more
> > portability between environments, eg dev, test, prod. Defining those once
> > and allowing for nifi to pull them in makes sense to me and I think is
> > common in container environments.
> >
> > Looking forward to discussing more and better approaches.
> > Chad
> >
> >> On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto  wrote:
> >>
> >> Hi Chad,
> >>
> >> Parameters were introduced as a way to deprecate (NiFi) variables
> >> entirely. I’m not sure that introducing a dependency between the two is a
> >> positive step forward. I think there is a separate conversation to be had
> >> about allowing parameters access to environment variables, but I think this
> >> could introduce problems as parameters are designed for flexibility and
> >> portability, and moving from a system where a parameter was actually a
> >> pass-through to an environment variable would cause unexpected problems on
> >> the destination system.
> >>
> >> I think the pros and cons of this need to be clearly enumerated and
> >> discussed here. Thanks for bringing this up.
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> alopresto.apa...@gmail.com
> >> He/Him
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
>  On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I was configuring an SSL Context Controller Service today and had the
> >>> keystores and passwords passed into the container via environment
> >>> variables. I thought it would be nice to be able to reference these from
> >>> the parameter context. Maybe either giving Parameter Context values the
> >>> VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
> >>> references external to nifi?
> >>>
> >>> I think for refreshing the Parameter Context on those external changes,
> >> it
> >>> would require an edit/re-apply just as it does now, and would have to
> >> make
> >>> sure it is well documented.
> >>>
> >>> I'd be interested in creating a PR for this if the idea makes sense and
> >> is
> >>> acceptable.
> >>>
> >>> Thanks,
> >>> Chad
> >>
> >>
>

Re: [VOTE] Release Apache NiFi Registry 0.8.0

[Bryan Bende]

+1 (binding)

- Verified everything in the standard release helper
- Ran the test-all-dbs profile, created one minor jira [1]
- Ran a secure registry with secure nifi
- Test basic save and import

Thanks for RM'ing!

[1] https://issues.apache.org/jira/browse/NIFIREG-426

On Fri, Oct 16, 2020 at 11:56 AM Andrew Lim  wrote:
>
> +1 (binding)
>
> -Ran full clean install on OS X (Catalina 10.15.2)
> -Tested secure NiFi Registry 0.8.0 with secure NiFi 1.12.1
> -Tested basic flow, bucket and user functionality
> -Tested common versioned flow use cases
> -Reviewed documentation updates
>
> Drew
>
> > On Oct 14, 2020, at 11:25 AM, Pierre Villard  
> > wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache NiFi
> > Registry 0.8.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1174
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.8.0/
> >
> > A helpful reminder on how the release candidate verification process works:
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-registry-0.8.0-RC1
> > The Git commit ID is 74c7435386b1118ed0b545efea7343deec6f4b29
> > https://gitbox.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=74c7435386b1118ed0b545efea7343deec6f4b29
> >
> > Checksums of nifi-registry-0.8.0-source-release.zip:
> > SHA256: afa4409a6b82aabd25fece787ba5eba1496313a3370bdf503c29d24f1d0b5a2c
> > SHA512:
> > 9c6e516f24b61b11dd889a5ea14950e7498801b7a51bdc14e4a8c091e9b834db213968bce8ee360882da9ee1c29aa8062eb6b9d9ec99c62b2f332c84aefe110f
> >
> > Release artifacts are signed with the key F92A93B30C07C6D5. The KEYS file
> > is available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 19 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12348583=12320920
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFIREG/Release+Notes#ReleaseNotes-NiFiRegistry0.8.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [ ] +1 Release this package as nifi-registry-0.8.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
>

Re: Dynamic PropertyDescriptors

[Bryan Bende]

Hello,

There is an open PR for "dependent properties" which will hopefully be
merged soon, but I think it is slightly different than what you are
asking for.

The idea is that some properties should only be shown based on how
another property is configured, i.e. changing a strategy property to
strategy1 should remove any properties relevant to other strategies so
the user doesn't have to see a bunch of stuff that isn't relevant.

In your scenario it isn't adding/removing whole properties, but
updating the values of a property. Not sure that has been considered
yet.

You basically have to implement customValidate and implement logic
that makes the processor invalid if your two properties have values
that shouldn't be used together.

-Bryan

On Wed, Oct 14, 2020 at 10:27 AM Jeremy Dyer  wrote:
>
> Hello,
>
> Is it possible to have valid values for a PropertyDescriptor change based
> on another selection?
>
> Ex: I have a ControllerService Property and if I change that
> ControllerService in the dropdown I would like the valid values for another
> Property to be changed in the UI.
>
> I'm thinking this isn't really possible since the logic occurs on the
> backend after "Apply" has been pressed. I know I can "Apply" and
> reconfigure to see the new values but would like something more immediate.
>
> Thanks!
> Jeremy Dyer

Re: [discuss] NiFi Registry 0.8.0 release

[Bryan Bende]

Thanks Pierre, let me know if you run into any issues.

On Tue, Oct 13, 2020 at 10:59 AM Kevin Doran  wrote:
>
> Great! Thanks everyone!
>
> > On Oct 13, 2020, at 10:42, Pierre Villard  
> > wrote:
> >
> > Thanks Bryan! Happy to give the RM process a try for this 0.8.0 release.
> >
> > Le mar. 13 oct. 2020 à 16:37, Joe Witt  a écrit :
> >
> >> sounds good and thanks.  happy to rm a nifi release too if needed.
> >>
> >> On Tue, Oct 13, 2020 at 7:33 AM Bryan Bende  wrote:
> >>
> >>> Would like to kick out registry 0.8.0 soon. Some nice improvements
> >>> since 0.7.0, including OIDC for registry. Plus a couple of changes on
> >>> NiFi side that need a new release of nifi-registry-client. I can be
> >>> the RM unless anyone else is interested.
> >>>
> >>>
> >>>
> >> https://issues.apache.org/jira/browse/NIFIREG-423?jql=project%20%3D%20NIFIREG%20AND%20fixVersion%20%3D%200.8.0
> >>>
> >>
>

[discuss] NiFi Registry 0.8.0 release

[Bryan Bende]

Would like to kick out registry 0.8.0 soon. Some nice improvements
since 0.7.0, including OIDC for registry. Plus a couple of changes on
NiFi side that need a new release of nifi-registry-client. I can be
the RM unless anyone else is interested.

https://issues.apache.org/jira/browse/NIFIREG-423?jql=project%20%3D%20NIFIREG%20AND%20fixVersion%20%3D%200.8.0

Re: Regarding Concurrent Tasks & Maximum Timer Driven Thread Count

[Bryan Bende]

Hello,

Mark Payne recently published a video which relates to this topic and
should answer your question:

https://www.youtube.com/watch?v=pZq0EbfDBy4

Thanks,

Bryan

On Fri, Sep 25, 2020 at 2:57 PM Midhun Mohan  wrote:
>
> Hi all,
>
> Lets say, If I want to configure my Nifi processor to have 100/ 200/ 500
> concurrent tasks, what should my
> Maximum Timer Driven Thread Count be?
>
> Also Should I need to take care of anything related to instance like, its
> ram or disk space,(if it is not deployed in a cluster to load balance), to
> make sure it is not being destroyed by my whole new setup.
>
>
> Or else
>
> If I can get a rule saying what all are depending on the having a larger
> count of concurrent tasks, then it would be helpful
>
> --
>
>
> Regards,
> Midhun Mohan

Re: ZooKeeper compatibility

[Bryan Bende]

Yes from 1.10.0 on it requires ZK 3.5.x.

On Tue, Sep 22, 2020 at 12:43 PM Mark Bean  wrote:
>
> The version of the embedded ZooKeeper was updated to 3.5.5 in NiFi 1.11.x.
> Does this have implications for running NiFi using an external ZooKeeper?
> Specifically, when using an external ZooKeeper, is version 3.5.5 required
> in order to run a NiFi 1.11.x cluster?
>
> Thanks.

[RESULT] [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.2

[Bryan Bende]

Apache NiFi Community,

I am pleased to announce that the  Apache NiFi NAR Maven Plugin 1.3.2
release passes with
4 +1 (binding) votes
0 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:
https://mail-archives.apache.org/mod_mbox/nifi-dev/202009.mbox/%3CCALo_M1_nTCD_zX8YQi7X8whakPd9rV1PCCh%2B%2BP-bmY0bd9Hb0w%40mail.gmail.com%3E

On Mon, Sep 21, 2020 at 4:18 PM Bryan Bende  wrote:
>
> +1 binding
>
> On Mon, Sep 21, 2020 at 12:56 PM Joe Witt  wrote:
> >
> > +1 (binding).
> >
> > Confirmed sig/hashes/commit is present and build works.
> >
> > Thanks Bryan!  Also FYI initially sig check failed due to expired cert.
> > But after reimporting latest keys all good again.  Thanks
> >
> > Joe
> >
> > On Fri, Sep 18, 2020 at 6:33 AM Pierre Villard 
> > wrote:
> >
> > > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.2
> > >
> > > Checked commit and hashes, looks good. Thanks Bryan!
> > >
> > > Le mer. 16 sept. 2020 à 19:10, Matt Burgess  a
> > > écrit :
> > >
> > > > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.2
> > > >
> > > > Verified Git commit and hashes, ran full NiFi build with this version
> > > > of the nar-maven-plugin, verified a simple flow.  Thanks for RM'ing
> > > > Bryan!
> > > >
> > > > On Wed, Sep 16, 2020 at 12:26 PM Bryan Bende  wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I am pleased to be calling this vote for the source release of Apache
> > > > > NiFi NAR Maven Plugin 1.3.2.
> > > > >
> > > > > The source zip, including signatures, digests, etc. can be found at:
> > > > > https://repository.apache.org/content/repositories/orgapachenifi-1168
> > > > >
> > > > > The source being voted upon and the convenience binaries can be found
> > > at:
> > > > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.2/
> > > > >
> > > > > A helpful reminder on how the release candidate verification process
> > > > works:
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > > > >
> > > > > The Git tag is nifi-nar-maven-plugin-1.3.2-RC1
> > > > > The Git commit ID is 9a4b78947b4633fb816b75f3afa9d7e334a40b66
> > > > >
> > > >
> > > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=9a4b78947b4633fb816b75f3afa9d7e334a40b66
> > > > >
> > > > > Checksums of nifi-nar-maven-plugin-1.3.2-source-release.zip:
> > > > > SHA256:
> > > f4374dce25b0ca75f2e95cce5c4d29474d9f10695a04baba2efc84ca5d361c2a
> > > > > SHA512:
> > > >
> > > 2b80c3c9c91f2ed7db16bc9fc568fee53751674492ad3fe9ffc9bed6dc5027a58c808fc7734d57819ffd025ab50c6807e84c6feaa576c5e6d02a7b17d20816dd
> > > > >
> > > > > Release artifacts are signed with the following key:
> > > > > https://people.apache.org/keys/committer/bbende.asc
> > > > >
> > > > > KEYS file available here:
> > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > > >
> > > > > 1 issues was closed/resolved for this release:
> > > > >
> > > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12345514
> > > > >
> > > > > Release note highlights can be found here:
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.2
> > > > >
> > > > > The vote will be open for 72 hours.
> > > > > Please download the release candidate and evaluate the necessary items
> > > > > including checking hashes, signatures, build
> > > > > from source, and test. Then please vote:
> > > > >
> > > > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.2
> > > > > [ ] +0 no opinion
> > > > > [ ] -1 Do not release this package because...
> > > >
> > >

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.2

[Bryan Bende]

+1 binding

On Mon, Sep 21, 2020 at 12:56 PM Joe Witt  wrote:
>
> +1 (binding).
>
> Confirmed sig/hashes/commit is present and build works.
>
> Thanks Bryan!  Also FYI initially sig check failed due to expired cert.
> But after reimporting latest keys all good again.  Thanks
>
> Joe
>
> On Fri, Sep 18, 2020 at 6:33 AM Pierre Villard 
> wrote:
>
> > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.2
> >
> > Checked commit and hashes, looks good. Thanks Bryan!
> >
> > Le mer. 16 sept. 2020 à 19:10, Matt Burgess  a
> > écrit :
> >
> > > +1 (binding) Release this package as nifi-nar-maven-plugin-1.3.2
> > >
> > > Verified Git commit and hashes, ran full NiFi build with this version
> > > of the nar-maven-plugin, verified a simple flow.  Thanks for RM'ing
> > > Bryan!
> > >
> > > On Wed, Sep 16, 2020 at 12:26 PM Bryan Bende  wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > > NiFi NAR Maven Plugin 1.3.2.
> > > >
> > > > The source zip, including signatures, digests, etc. can be found at:
> > > > https://repository.apache.org/content/repositories/orgapachenifi-1168
> > > >
> > > > The source being voted upon and the convenience binaries can be found
> > at:
> > > >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.2/
> > > >
> > > > A helpful reminder on how the release candidate verification process
> > > works:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > > >
> > > > The Git tag is nifi-nar-maven-plugin-1.3.2-RC1
> > > > The Git commit ID is 9a4b78947b4633fb816b75f3afa9d7e334a40b66
> > > >
> > >
> > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=9a4b78947b4633fb816b75f3afa9d7e334a40b66
> > > >
> > > > Checksums of nifi-nar-maven-plugin-1.3.2-source-release.zip:
> > > > SHA256:
> > f4374dce25b0ca75f2e95cce5c4d29474d9f10695a04baba2efc84ca5d361c2a
> > > > SHA512:
> > >
> > 2b80c3c9c91f2ed7db16bc9fc568fee53751674492ad3fe9ffc9bed6dc5027a58c808fc7734d57819ffd025ab50c6807e84c6feaa576c5e6d02a7b17d20816dd
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/bbende.asc
> > > >
> > > > KEYS file available here:
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 1 issues was closed/resolved for this release:
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12345514
> > > >
> > > > Release note highlights can be found here:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.2
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary items
> > > > including checking hashes, signatures, build
> > > > from source, and test. Then please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.2
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because...
> > >
> >

Release Helper for Apache NiFi NAR Maven Plugin 1.3.2 RC1

[Bryan Bende]

In addition to the standard wiki pages for release verification [1], I
put together a small example bundle that demonstrates the issue [2].

The master branch is using the 1.3.1 version of the plugin and will
fail to build. The "fix" branch is using 1.3.2 and should succeed. You
have to first build the source for 1.3.2 using "mvn clean install" so
that it is available in your local repo.

Thanks,

Bryan

[1] 
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
[2] https://github.com/bbende/nifi-nar-error-bundle

[VOTE] Release Apache NiFi NAR Maven Plugin 1.3.2

[Bryan Bende]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi NAR Maven Plugin 1.3.2.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1168

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.2/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-nar-maven-plugin-1.3.2-RC1
The Git commit ID is 9a4b78947b4633fb816b75f3afa9d7e334a40b66
https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=9a4b78947b4633fb816b75f3afa9d7e334a40b66

Checksums of nifi-nar-maven-plugin-1.3.2-source-release.zip:
SHA256: f4374dce25b0ca75f2e95cce5c4d29474d9f10695a04baba2efc84ca5d361c2a
SHA512: 
2b80c3c9c91f2ed7db16bc9fc568fee53751674492ad3fe9ffc9bed6dc5027a58c808fc7734d57819ffd025ab50c6807e84c6feaa576c5e6d02a7b17d20816dd

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/bbende.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

1 issues was closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12345514

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.2

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-nar-maven-plugin-1.3.2
[ ] +0 no opinion
[ ] -1 Do not release this package because...