Re: setting up secure nifi

[Matt Burgess]

They work for me, perhaps there was a connectivity issue or something?

On Thu, Feb 1, 2018 at 10:56 AM, Anil Rai  wrote:
> The below links does not work. Have they moved somewhere else?
>
> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
> https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html
> https://blog.rosander.ninja/nifi/toolkit/tls/2016/
> 09/20/tls-toolkit-standalone-multi.html
>
> Thanks
> Anil
>
> On Thu, Feb 1, 2018 at 10:35 AM, Anil Rai  wrote:
>
>> Thanks Andy. It did resolve my issue. I got it working.
>> Thanks again for all the links. Very helpful.
>>
>> Cheers
>> Anil
>>
>>
>> On Thu, Feb 1, 2018 at 10:14 AM, Andy LoPresto 
>> wrote:
>>
>>> Hi Anil,
>>>
>>> In addition to Bryan’s explanation, there are a number of blog posts and
>>> articles covering this topic:
>>>
>>> * Authorization and Multi-Tenancy by Bryan Bende [1]
>>> * Secured Cluster Setup by Pierre Villard [2]
>>> * TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
>>> * Initial Admin Identity section of Apache NiFi Admin Guide [4]
>>> * Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
>>> * Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan
>>> Rosander [6]
>>>
>>> The sequence “dc=example,dc=com” in your current user DN (Distinguished
>>> Name) is incorrect and not present in the DN of the certificate. I imagine
>>> you copied this from an example posted online. “dc=“ is a sequence used in
>>> DNS to indicate “Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is
>>> the RDN (Relative Distinguished Name) of your user, and “dc=example,dc=com”
>>> would be the parent DN. But when you generated the certificate, you did not
>>> provide this information, so the DNs do not match, and NiFi correctly
>>> asserts that this is not a valid certificate identifying the user DN you
>>> specified in your XML files. Removing “dc=example,dc=com” from that
>>> definition as Bryan suggested will resolve your issue.
>>>
>>> [1] https://bryanbende.com/development/2016/08/17/apache-nif
>>> i-1-0-0-authorization-and-multi-tenancy
>>> [2] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-s
>>> ecured-cluster-setup/
>>> [3] https://nifi.apache.org/docs/nifi-docs/html/administrati
>>> on-guide.html#tls-generation-toolkit
>>> [4] https://nifi.apache.org/docs/nifi-docs/html/administrati
>>> on-guide.html#initial-admin-identity
>>> [5] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/
>>> tls-toolkit-intro.html
>>> [6] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/
>>> tls-toolkit-standalone-multi.html
>>> [7] https://en.wikipedia.org/wiki/Lightweight_Directory_Acce
>>> ss_Protocol#Directory_structure
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> *alopresto.apa...@gmail.com *
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Jan 31, 2018, at 7:32 PM, Bryan Bende  wrote:
>>>
>>> It’s the same problem, your initial admin should be:
>>>
>>> CN=TC, OU=NIFI
>>>
>>> Not
>>>
>>> CN=TC,OU=NIFI,dc=example,dc=com
>>>
>>> The first one is the DN of your client cert, the second one is not.
>>>
>>> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  wrote:
>>>
>>> Hi Bryan,
>>>
>>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>>> same error.
>>> Now the entry looks like
>>>CN=TC,OU=NIFI,dc=example,
>>> dc=com
>>>
>>> Also what does dc stand for after CN and OU. Is that a problem.
>>> Is there a blog that talks about installing and making it https using
>>> toolkit?. I did not find any good post that talks end to end from
>>> installing to making it secure using tls toolkit.
>>>
>>> Any help is appreciated.
>>>
>>> Thanks
>>> Anil
>>>
>>>
>>>
>>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende  wrote:
>>>
>>> Hello,
>>>
>>> The identity in authorizers.xml for your initial admin does not match the
>>> identity of your client cert.
>>>
>>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>>>
>>> is
>>>
>>> the DN of your client cert.
>>>
>>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>>> authorizations.xml, and start back up.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai  wrote:
>>>
>>> All,
>>>
>>> I am trying to install nifi 1.5 and making it https. Below is the steps
>>> followed and the error i am getting. Below is the config and log files
>>> content. Please help
>>>
>>> 1. Installed nifi 1.5
>>> 2. Installed nifi toolkit 1.5
>>> 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
>>> 'CN=TC,OU=NIFI' -O -o ../security_output
>>> 4. Copied generated keystore, truststore and nifi properties to
>>>
>>> nifi/config
>>>
>>> folder
>>> 5. Imported the generated certificate to chrome browser
>>> 6. 

RE: [EXT] Re: Double click for failure queues

[Karthik Kothareddy (karthikk) [CONT - Type 2]]

I mean on the connection label itself not on the path which bends it. 
Specifically, I observed this behavior on the relationships that are routed 
back to the same processor.

 
-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com] 
Sent: Thursday, February 01, 2018 12:46 PM
To: dev@nifi.apache.org
Subject: [EXT] Re: Double click for failure queues

There is no notion of a 'failure queue' versus any other queue.
Processors have named relationships that once connected to another thing forms 
a connection.  They have no special meaning.

So, you're saying that some connections you double click bring up the dialogue 
and others do not respond to a double click?

On Thu, Feb 1, 2018 at 2:02 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] 
 wrote:
> All,
>
> I am using 1.4.0 and I see that double-click to configure (for both queues 
> and processors) feature as very useful and intuitive to use. However, if I 
> double click on a queue which has a failure relationship it doesn't work as 
> expected and I have to right-click to configure it. Is this by design? Or a 
> bug in UI?
>
> Just curious...
>
> Thanks
> Karthik

Re: Double click for failure queues

[Matt Gilman]

Double-clicking on the connection label should open the connection
configuration dialog. Double-clicking on the actual path will generate a
bend point. The connection label can then be attached to any bend point.

Matt

On Thu, Feb 1, 2018 at 2:02 PM, Karthik Kothareddy (karthikk) [CONT - Type
2]  wrote:

> All,
>
> I am using 1.4.0 and I see that double-click to configure (for both queues
> and processors) feature as very useful and intuitive to use. However, if I
> double click on a queue which has a failure relationship it doesn't work as
> expected and I have to right-click to configure it. Is this by design? Or a
> bug in UI?
>
> Just curious...
>
> Thanks
> Karthik
>

Re: Double click for failure queues

[Joe Witt]

There is no notion of a 'failure queue' versus any other queue.
Processors have named relationships that once connected to another
thing forms a connection.  They have no special meaning.

So, you're saying that some connections you double click bring up the
dialogue and others do not respond to a double click?

On Thu, Feb 1, 2018 at 2:02 PM, Karthik Kothareddy (karthikk) [CONT -
Type 2]  wrote:
> All,
>
> I am using 1.4.0 and I see that double-click to configure (for both queues 
> and processors) feature as very useful and intuitive to use. However, if I 
> double click on a queue which has a failure relationship it doesn't work as 
> expected and I have to right-click to configure it. Is this by design? Or a 
> bug in UI?
>
> Just curious...
>
> Thanks
> Karthik

Double click for failure queues

[Karthik Kothareddy (karthikk) [CONT - Type 2]]

All,

I am using 1.4.0 and I see that double-click to configure (for both queues and 
processors) feature as very useful and intuitive to use. However, if I double 
click on a queue which has a failure relationship it doesn't work as expected 
and I have to right-click to configure it. Is this by design? Or a bug in UI?

Just curious...

Thanks
Karthik

Re: Incorrect return classes specified on api annotations

[Matt Gilman]

Thanks for reaching out Charlie. If you wanted to submit a PR for this, I
would be happy to review/merge it for you.

Matt

On Wed, Jan 31, 2018 at 5:34 PM, Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:

> Hi,
>
> I opened up https://issues.apache.org/jira/browse/NIFI-4835 earlier today
> for a few places where the return type class did not reflect the entity
> that is being returned. For now, I've locally patched my build and was able
> to regenerate a correct swagger json, but figured id bring it up here as
> well so it can hopefully make it into the next release.
>
> Thanks!
>

Re: setting up secure nifi

[Anil Rai]

The below links does not work. Have they moved somewhere else?

https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html
https://blog.rosander.ninja/nifi/toolkit/tls/2016/
09/20/tls-toolkit-standalone-multi.html

Thanks
Anil

On Thu, Feb 1, 2018 at 10:35 AM, Anil Rai  wrote:

> Thanks Andy. It did resolve my issue. I got it working.
> Thanks again for all the links. Very helpful.
>
> Cheers
> Anil
>
>
> On Thu, Feb 1, 2018 at 10:14 AM, Andy LoPresto 
> wrote:
>
>> Hi Anil,
>>
>> In addition to Bryan’s explanation, there are a number of blog posts and
>> articles covering this topic:
>>
>> * Authorization and Multi-Tenancy by Bryan Bende [1]
>> * Secured Cluster Setup by Pierre Villard [2]
>> * TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
>> * Initial Admin Identity section of Apache NiFi Admin Guide [4]
>> * Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
>> * Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan
>> Rosander [6]
>>
>> The sequence “dc=example,dc=com” in your current user DN (Distinguished
>> Name) is incorrect and not present in the DN of the certificate. I imagine
>> you copied this from an example posted online. “dc=“ is a sequence used in
>> DNS to indicate “Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is
>> the RDN (Relative Distinguished Name) of your user, and “dc=example,dc=com”
>> would be the parent DN. But when you generated the certificate, you did not
>> provide this information, so the DNs do not match, and NiFi correctly
>> asserts that this is not a valid certificate identifying the user DN you
>> specified in your XML files. Removing “dc=example,dc=com” from that
>> definition as Bryan suggested will resolve your issue.
>>
>> [1] https://bryanbende.com/development/2016/08/17/apache-nif
>> i-1-0-0-authorization-and-multi-tenancy
>> [2] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-s
>> ecured-cluster-setup/
>> [3] https://nifi.apache.org/docs/nifi-docs/html/administrati
>> on-guide.html#tls-generation-toolkit
>> [4] https://nifi.apache.org/docs/nifi-docs/html/administrati
>> on-guide.html#initial-admin-identity
>> [5] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/
>> tls-toolkit-intro.html
>> [6] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/
>> tls-toolkit-standalone-multi.html
>> [7] https://en.wikipedia.org/wiki/Lightweight_Directory_Acce
>> ss_Protocol#Directory_structure
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jan 31, 2018, at 7:32 PM, Bryan Bende  wrote:
>>
>> It’s the same problem, your initial admin should be:
>>
>> CN=TC, OU=NIFI
>>
>> Not
>>
>> CN=TC,OU=NIFI,dc=example,dc=com
>>
>> The first one is the DN of your client cert, the second one is not.
>>
>> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  wrote:
>>
>> Hi Bryan,
>>
>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>> same error.
>> Now the entry looks like
>>CN=TC,OU=NIFI,dc=example,
>> dc=com
>>
>> Also what does dc stand for after CN and OU. Is that a problem.
>> Is there a blog that talks about installing and making it https using
>> toolkit?. I did not find any good post that talks end to end from
>> installing to making it secure using tls toolkit.
>>
>> Any help is appreciated.
>>
>> Thanks
>> Anil
>>
>>
>>
>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende  wrote:
>>
>> Hello,
>>
>> The identity in authorizers.xml for your initial admin does not match the
>> identity of your client cert.
>>
>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>>
>> is
>>
>> the DN of your client cert.
>>
>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>> authorizations.xml, and start back up.
>>
>> Thanks,
>>
>> Bryan
>>
>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai  wrote:
>>
>> All,
>>
>> I am trying to install nifi 1.5 and making it https. Below is the steps
>> followed and the error i am getting. Below is the config and log files
>> content. Please help
>>
>> 1. Installed nifi 1.5
>> 2. Installed nifi toolkit 1.5
>> 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
>> 'CN=TC,OU=NIFI' -O -o ../security_output
>> 4. Copied generated keystore, truststore and nifi properties to
>>
>> nifi/config
>>
>> folder
>> 5. Imported the generated certificate to chrome browser
>> 6. Modified authorizers.xml as attached.
>> 7. With required restarts. Now when i enter the below url in the
>>
>> browser, I
>>
>> see the below error.
>>
>> https://localhost:9443/nifi/
>>
>> Insufficient Permissions
>>
>>   - home
>>
>> Unknown user with identity 'CN=TC, OU=NIFI'. Contact the 

Re: [DISCUSS] Addressing Lingering Pull Requests

[Matt Burgess]

I agree with Andy on the Docker point, I think it could be too high a
barrier for contribution in some cases. However I think we can build
out / extend a "new component" section of the PR template that has
more best practices, recommendations, suggestions.

I like the idea of the reviewer providing a sample template, perhaps
via Gist or something, but that should probably start as a
recommendation and not a requirement. If a reviewer is unfamiliar with
the new component (or its third-party libraries), they can always ask
the author if they have a test flow they can share, and hopefully the
practice will become status quo organically.

Another thing we could add is checkbox(es) around standard efforts
associated with the addition of new components/bundles. Specifically
if you add new NARs, you have to add them to the nifi-assembly POM and
their versions to the top-level POM, not to mention the bundle to the
nifi-nar-bundles POM.

Regards,
Matt


On Thu, Feb 1, 2018 at 10:20 AM, Andy LoPresto  wrote:
> Mike,
>
> I think that would be awesome for reviewers (and that is where most of my
> time is spent on the review side), but I also think that sets a really high
> bar for contribution. Many of the users who submit pull requests are
> first-time contributors or new to the project, and I believe the easier we
> can make contributing, the more we will grow and strengthen our community.
> Someone might have experience with a weird protocol or multithreading and be
> able to offer expertise there, but not have the familiarity with Docker. For
> experienced users contributing advanced functionality though, I think it
> would be excellent and definitely streamline the review process.
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 1, 2018, at 9:31 AM, Mike Thomsen  wrote:
>
> It looks like there are at least 10 new processors and services in the
> backlog, and quite a few modifications to existing ones.
>
> Something I think would really help here is to expand the scope of
> requirements for submitting a new processor for code review to include:
>
> 1. docker-compose file that sets up a complete development environment w/
> appropriate port mappings that just work when used with NiFi running
> outside of Docker on the reviewer's machine.
>
> 2. A sample flow, a really KISS example w/ GenerateFlowFile or something
> that spits out a query or sample that can let the reviewer really see the
> submitter's idea of what the input should look like.
>
> Whether those are stored on a Wiki or in the Git repo doesn't really
> matter. I think submitting those artifacts will really reduce the burden of
> quickly going from "LGTM, JUnits seem to run" to "OK, I see it actually
> running as expected."
>
> On Tue, Jan 30, 2018 at 12:38 AM, James Wing  wrote:
>
> This is a great idea, Mark, thanks for proposing it.  30 days after last
> review comment seems like a good, enforceable standard.
>
> James
>
> On Mon, Jan 29, 2018 at 8:29 AM, Mark Payne  wrote:
>
> All,
>
> We do from time to time go through the backlog of PR's that need to be
> reviewed and
> start a "cleansing" process, closing out any old PR's that appear to have
> stalled out.
> When we do this, though, we typically will start sending out e-mails
> asking if there are
> any stalled PR's that we shouldn't close and start trying to decipher
> which ones are okay
> to close out and which ones are not. This puts quite an onus on the
> committer who is
> trying to clean this up. It also can result in having a large number of
> outstanding Pull Requests,
> which I believe makes the community look bad because it gives the
> appearance that we are
> not doing a good job of being responsive to Pull Requests that are
> submitted.
>
> I would like to propose that we set a new "standard" that is: if we have
> any Pull Request
> that has been stalled (and by "stalled" I mean a committer has reviewed
> the PR and did
> not merge but asked for clarifications or modifications and the
> contributor has not pushed
> any new commit or responded to the comments) for at least 30 days, that
>
> we
>
> go ahead
> and close the Pull Request (after commenting on the PR that it is being
> closed due to a lack
> of activity and that the contributor is more than welcome to open a new
>
> PR
>
> if necessary).
>
> I feel like this gives contributors enough time to address concerns and
>
> it
>
> is simple enough
> to create a new Pull Request if the need arises. Alternatively, if the
> contributor realizes that
> they need more time, they can simply comment on the PR that they are
>
> still
>
> interested in
> working on it but just need more time, and the simple act of commenting
> will mean that the
> PR is no longer stalled, as defined above.
>
> Any thoughts on such a proposal? Any better alternatives 

Re: setting up secure nifi

[Anil Rai]

Thanks Andy. It did resolve my issue. I got it working.
Thanks again for all the links. Very helpful.

Cheers
Anil


On Thu, Feb 1, 2018 at 10:14 AM, Andy LoPresto  wrote:

> Hi Anil,
>
> In addition to Bryan’s explanation, there are a number of blog posts and
> articles covering this topic:
>
> * Authorization and Multi-Tenancy by Bryan Bende [1]
> * Secured Cluster Setup by Pierre Villard [2]
> * TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
> * Initial Admin Identity section of Apache NiFi Admin Guide [4]
> * Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
> * Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan
> Rosander [6]
>
> The sequence “dc=example,dc=com” in your current user DN (Distinguished
> Name) is incorrect and not present in the DN of the certificate. I imagine
> you copied this from an example posted online. “dc=“ is a sequence used in
> DNS to indicate “Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is
> the RDN (Relative Distinguished Name) of your user, and “dc=example,dc=com”
> would be the parent DN. But when you generated the certificate, you did not
> provide this information, so the DNs do not match, and NiFi correctly
> asserts that this is not a valid certificate identifying the user DN you
> specified in your XML files. Removing “dc=example,dc=com” from that
> definition as Bryan suggested will resolve your issue.
>
> [1] https://bryanbende.com/development/2016/08/17/apache-
> nifi-1-0-0-authorization-and-multi-tenancy
> [2] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-
> secured-cluster-setup/
> [3] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
> [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> initial-admin-identity
> [5] https://blog.rosander.ninja/nifi/toolkit/tls/2016/
> 09/19/tls-toolkit-intro.html
> [6] https://blog.rosander.ninja/nifi/toolkit/tls/2016/
> 09/20/tls-toolkit-standalone-multi.html
> [7] https://en.wikipedia.org/wiki/Lightweight_Directory_
> Access_Protocol#Directory_structure
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 31, 2018, at 7:32 PM, Bryan Bende  wrote:
>
> It’s the same problem, your initial admin should be:
>
> CN=TC, OU=NIFI
>
> Not
>
> CN=TC,OU=NIFI,dc=example,dc=com
>
> The first one is the DN of your client cert, the second one is not.
>
> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  wrote:
>
> Hi Bryan,
>
> Thanks for the quick reply. I did followed your steps. But I am seeing the
> same error.
> Now the entry looks like
>CN=TC,OU=NIFI,dc=example,
> dc=com
>
> Also what does dc stand for after CN and OU. Is that a problem.
> Is there a blog that talks about installing and making it https using
> toolkit?. I did not find any good post that talks end to end from
> installing to making it secure using tls toolkit.
>
> Any help is appreciated.
>
> Thanks
> Anil
>
>
>
> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende  wrote:
>
> Hello,
>
> The identity in authorizers.xml for your initial admin does not match the
> identity of your client cert.
>
> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>
> is
>
> the DN of your client cert.
>
> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
> authorizations.xml, and start back up.
>
> Thanks,
>
> Bryan
>
> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai  wrote:
>
> All,
>
> I am trying to install nifi 1.5 and making it https. Below is the steps
> followed and the error i am getting. Below is the config and log files
> content. Please help
>
> 1. Installed nifi 1.5
> 2. Installed nifi toolkit 1.5
> 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
> 'CN=TC,OU=NIFI' -O -o ../security_output
> 4. Copied generated keystore, truststore and nifi properties to
>
> nifi/config
>
> folder
> 5. Imported the generated certificate to chrome browser
> 6. Modified authorizers.xml as attached.
> 7. With required restarts. Now when i enter the below url in the
>
> browser, I
>
> see the below error.
>
> https://localhost:9443/nifi/
>
> Insufficient Permissions
>
>   - home
>
> Unknown user with identity 'CN=TC, OU=NIFI'. Contact the system
> administrator.
>
>
> authorizers.xml
> 
>
>file-user-group-provider
>org.apache.nifi.authorization.
>
> FileUserGroupProvider
>
>./conf/users.xml
>
>
>cn=TC,ou=NIFI,dc=example,dc=com
>
>
>
>file-access-policy-provider
>
> org.apache.nifi.authorization.FileAccessPolicyProvider
>file-user-group-provider
>./conf/authorizations.xml
>cn=TC,ou=NIFI,dc=example,dc=com
>
>
>
>
> 
>
> nifi-user.log
> 

Re: [DISCUSS] Addressing Lingering Pull Requests

[Andy LoPresto]

Mike,

I think that would be awesome for reviewers (and that is where most of my time 
is spent on the review side), but I also think that sets a really high bar for 
contribution. Many of the users who submit pull requests are first-time 
contributors or new to the project, and I believe the easier we can make 
contributing, the more we will grow and strengthen our community. Someone might 
have experience with a weird protocol or multithreading and be able to offer 
expertise there, but not have the familiarity with Docker. For experienced 
users contributing advanced functionality though, I think it would be excellent 
and definitely streamline the review process.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 1, 2018, at 9:31 AM, Mike Thomsen  wrote:
> 
> It looks like there are at least 10 new processors and services in the
> backlog, and quite a few modifications to existing ones.
> 
> Something I think would really help here is to expand the scope of
> requirements for submitting a new processor for code review to include:
> 
> 1. docker-compose file that sets up a complete development environment w/
> appropriate port mappings that just work when used with NiFi running
> outside of Docker on the reviewer's machine.
> 
> 2. A sample flow, a really KISS example w/ GenerateFlowFile or something
> that spits out a query or sample that can let the reviewer really see the
> submitter's idea of what the input should look like.
> 
> Whether those are stored on a Wiki or in the Git repo doesn't really
> matter. I think submitting those artifacts will really reduce the burden of
> quickly going from "LGTM, JUnits seem to run" to "OK, I see it actually
> running as expected."
> 
> On Tue, Jan 30, 2018 at 12:38 AM, James Wing  wrote:
> 
>> This is a great idea, Mark, thanks for proposing it.  30 days after last
>> review comment seems like a good, enforceable standard.
>> 
>> James
>> 
>> On Mon, Jan 29, 2018 at 8:29 AM, Mark Payne  wrote:
>> 
>>> All,
>>> 
>>> We do from time to time go through the backlog of PR's that need to be
>>> reviewed and
>>> start a "cleansing" process, closing out any old PR's that appear to have
>>> stalled out.
>>> When we do this, though, we typically will start sending out e-mails
>>> asking if there are
>>> any stalled PR's that we shouldn't close and start trying to decipher
>>> which ones are okay
>>> to close out and which ones are not. This puts quite an onus on the
>>> committer who is
>>> trying to clean this up. It also can result in having a large number of
>>> outstanding Pull Requests,
>>> which I believe makes the community look bad because it gives the
>>> appearance that we are
>>> not doing a good job of being responsive to Pull Requests that are
>>> submitted.
>>> 
>>> I would like to propose that we set a new "standard" that is: if we have
>>> any Pull Request
>>> that has been stalled (and by "stalled" I mean a committer has reviewed
>>> the PR and did
>>> not merge but asked for clarifications or modifications and the
>>> contributor has not pushed
>>> any new commit or responded to the comments) for at least 30 days, that
>> we
>>> go ahead
>>> and close the Pull Request (after commenting on the PR that it is being
>>> closed due to a lack
>>> of activity and that the contributor is more than welcome to open a new
>> PR
>>> if necessary).
>>> 
>>> I feel like this gives contributors enough time to address concerns and
>> it
>>> is simple enough
>>> to create a new Pull Request if the need arises. Alternatively, if the
>>> contributor realizes that
>>> they need more time, they can simply comment on the PR that they are
>> still
>>> interested in
>>> working on it but just need more time, and the simple act of commenting
>>> will mean that the
>>> PR is no longer stalled, as defined above.
>>> 
>>> Any thoughts on such a proposal? Any better alternatives that people have
>>> in mind?
>>> 
>>> Thanks
>>> -Mark
>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: setting up secure nifi

[Andy LoPresto]

Hi Anil,

In addition to Bryan’s explanation, there are a number of blog posts and 
articles covering this topic:

* Authorization and Multi-Tenancy by Bryan Bende [1]
* Secured Cluster Setup by Pierre Villard [2]
* TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
* Initial Admin Identity section of Apache NiFi Admin Guide [4]
* Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
* Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan Rosander [6]

The sequence “dc=example,dc=com” in your current user DN (Distinguished Name) 
is incorrect and not present in the DN of the certificate. I imagine you copied 
this from an example posted online. “dc=“ is a sequence used in DNS to indicate 
“Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is the RDN (Relative 
Distinguished Name) of your user, and “dc=example,dc=com” would be the parent 
DN. But when you generated the certificate, you did not provide this 
information, so the DNs do not match, and NiFi correctly asserts that this is 
not a valid certificate identifying the user DN you specified in your XML 
files. Removing “dc=example,dc=com” from that definition as Bryan suggested 
will resolve your issue.

[1] 
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
 

[2] 
https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ 

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
 

[4] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity
 

[5] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html 

[6] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/tls-toolkit-standalone-multi.html
 

[7] 
https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure
 


Andy LoPresto
alopre...@apache.org 
alopresto.apa...@gmail.com 
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 31, 2018, at 7:32 PM, Bryan Bende  > wrote:
> 
> It’s the same problem, your initial admin should be:
> 
> CN=TC, OU=NIFI
> 
> Not
> 
> CN=TC,OU=NIFI,dc=example,dc=com
> 
> The first one is the DN of your client cert, the second one is not.
> 
> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  > wrote:
> 
>> Hi Bryan,
>> 
>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>> same error.
>> Now the entry looks like
>>CN=TC,OU=NIFI,dc=example,
>> dc=com
>> 
>> Also what does dc stand for after CN and OU. Is that a problem.
>> Is there a blog that talks about installing and making it https using
>> toolkit?. I did not find any good post that talks end to end from
>> installing to making it secure using tls toolkit.
>> 
>> Any help is appreciated.
>> 
>> Thanks
>> Anil
>> 
>> 
>> 
>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende > > wrote:
>> 
>>> Hello,
>>> 
>>> The identity in authorizers.xml for your initial admin does not match the
>>> identity of your client cert.
>>> 
>>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>> is
>>> the DN of your client cert.
>>> 
>>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>>> authorizations.xml, and start back up.
>>> 
>>> Thanks,
>>> 
>>> Bryan
>>> 
>>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai >> > wrote:
>>> 
 All,
 
 I am trying to install nifi 1.5 and making it https. Below is the steps
 followed and the error i am getting. Below is the config and log files
 content. Please help
 
 1. Installed nifi 1.5
 2. Installed nifi toolkit 1.5
 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
 'CN=TC,OU=NIFI' -O -o ../security_output
 4. Copied generated keystore, truststore and nifi properties to
>>> nifi/config
 folder
 5. Imported the generated certificate to chrome browser
 6. Modified authorizers.xml as attached.
 7. With required restarts. Now when i enter the below url in the
>>> browser, I
 see the below error.
 
 https://localhost:9443/nifi/ 

Re: setting up secure nifi

[Andy LoPresto]

Hi Anil,

In addition to Bryan’s explanation, there are a number of blog posts and 
articles covering this topic:

* Authorization and Multi-Tenancy by Bryan Bende [1]
* Secured Cluster Setup by Pierre Villard [2]
* TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
* Initial Admin Identity section of Apache NiFi Admin Guide [4]
* Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
* Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan Rosander [6]

The sequence “dc=example,dc=com” in your current user DN (Distinguished Name) 
is incorrect and not present in the DN of the certificate. I imagine you copied 
this from an example posted online. “dc=“ is a sequence used in DNS to indicate 
“Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is the RDN (Relative 
Distinguished Name) of your user, and “dc=example,dc=com” would be the parent 
DN. But when you generated the certificate, you did not provide this 
information, so the DNs do not match, and NiFi correctly asserts that this is 
not a valid certificate identifying the user DN you specified in your XML 
files. Removing “dc=example,dc=com” from that definition as Bryan suggested 
will resolve your issue.

[1] 
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
 

[2] 
https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ 

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
 

[4] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity
 

[5] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html 

[6] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/tls-toolkit-standalone-multi.html
 

[7] 
https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure
 


Andy LoPresto
alopre...@apache.org 
alopresto.apa...@gmail.com 
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 31, 2018, at 7:32 PM, Bryan Bende  > wrote:
> 
> It’s the same problem, your initial admin should be:
> 
> CN=TC, OU=NIFI
> 
> Not
> 
> CN=TC,OU=NIFI,dc=example,dc=com
> 
> The first one is the DN of your client cert, the second one is not.
> 
> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  > wrote:
> 
>> Hi Bryan,
>> 
>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>> same error.
>> Now the entry looks like
>>CN=TC,OU=NIFI,dc=example,
>> dc=com
>> 
>> Also what does dc stand for after CN and OU. Is that a problem.
>> Is there a blog that talks about installing and making it https using
>> toolkit?. I did not find any good post that talks end to end from
>> installing to making it secure using tls toolkit.
>> 
>> Any help is appreciated.
>> 
>> Thanks
>> Anil
>> 
>> 
>> 
>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende > > wrote:
>> 
>>> Hello,
>>> 
>>> The identity in authorizers.xml for your initial admin does not match the
>>> identity of your client cert.
>>> 
>>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>> is
>>> the DN of your client cert.
>>> 
>>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>>> authorizations.xml, and start back up.
>>> 
>>> Thanks,
>>> 
>>> Bryan
>>> 
>>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai >> > wrote:
>>> 
 All,
 
 I am trying to install nifi 1.5 and making it https. Below is the steps
 followed and the error i am getting. Below is the config and log files
 content. Please help
 
 1. Installed nifi 1.5
 2. Installed nifi toolkit 1.5
 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
 'CN=TC,OU=NIFI' -O -o ../security_output
 4. Copied generated keystore, truststore and nifi properties to
>>> nifi/config
 folder
 5. Imported the generated certificate to chrome browser
 6. Modified authorizers.xml as attached.
 7. With required restarts. Now when i enter the below url in the
>>> browser, I
 see the below error.
 
 https://localhost:9443/nifi/ 

Re: setting up secure nifi

[Andy LoPresto]

Hi Anil,

In addition to Bryan’s explanation, there are a number of blog posts and 
articles covering this topic:

* Authorization and Multi-Tenancy by Bryan Bende [1]
* Secured Cluster Setup by Pierre Villard [2]
* TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
* Initial Admin Identity section of Apache NiFi Admin Guide [4]
* Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
* Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan Rosander [6]

The sequence “dc=example,dc=com” in your current user DN (Distinguished Name) 
is incorrect and not present in the DN of the certificate. I imagine you copied 
this from an example posted online. “dc=“ is a sequence used in DNS to indicate 
“Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is the RDN (Relative 
Distinguished Name) of your user, and “dc=example,dc=com” would be the parent 
DN. But when you generated the certificate, you did not provide this 
information, so the DNs do not match, and NiFi correctly asserts that this is 
not a valid certificate identifying the user DN you specified in your XML 
files. Removing “dc=example,dc=com” from that definition as Bryan suggested 
will resolve your issue.

[1] 
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
 

[2] 
https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ 

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
 

[4] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity
 

[5] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html 

[6] 
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/tls-toolkit-standalone-multi.html
 

[7] 
https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure
 


Andy LoPresto
alopre...@apache.org 
alopresto.apa...@gmail.com 
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 31, 2018, at 7:32 PM, Bryan Bende  > wrote:
> 
> It’s the same problem, your initial admin should be:
> 
> CN=TC, OU=NIFI
> 
> Not
> 
> CN=TC,OU=NIFI,dc=example,dc=com
> 
> The first one is the DN of your client cert, the second one is not.
> 
> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai  > wrote:
> 
>> Hi Bryan,
>> 
>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>> same error.
>> Now the entry looks like
>>CN=TC,OU=NIFI,dc=example,
>> dc=com
>> 
>> Also what does dc stand for after CN and OU. Is that a problem.
>> Is there a blog that talks about installing and making it https using
>> toolkit?. I did not find any good post that talks end to end from
>> installing to making it secure using tls toolkit.
>> 
>> Any help is appreciated.
>> 
>> Thanks
>> Anil
>> 
>> 
>> 
>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende > > wrote:
>> 
>>> Hello,
>>> 
>>> The identity in authorizers.xml for your initial admin does not match the
>>> identity of your client cert.
>>> 
>>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>> is
>>> the DN of your client cert.
>>> 
>>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>>> authorizations.xml, and start back up.
>>> 
>>> Thanks,
>>> 
>>> Bryan
>>> 
>>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai >> > wrote:
>>> 
 All,
 
 I am trying to install nifi 1.5 and making it https. Below is the steps
 followed and the error i am getting. Below is the config and log files
 content. Please help
 
 1. Installed nifi 1.5
 2. Installed nifi toolkit 1.5
 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
 'CN=TC,OU=NIFI' -O -o ../security_output
 4. Copied generated keystore, truststore and nifi properties to
>>> nifi/config
 folder
 5. Imported the generated certificate to chrome browser
 6. Modified authorizers.xml as attached.
 7. With required restarts. Now when i enter the below url in the
>>> browser, I
 see the below error.
 
 https://localhost:9443/nifi/ 

Re: [DISCUSS] Addressing Lingering Pull Requests

[Mike Thomsen]

It looks like there are at least 10 new processors and services in the
backlog, and quite a few modifications to existing ones.

Something I think would really help here is to expand the scope of
requirements for submitting a new processor for code review to include:

1. docker-compose file that sets up a complete development environment w/
appropriate port mappings that just work when used with NiFi running
outside of Docker on the reviewer's machine.

2. A sample flow, a really KISS example w/ GenerateFlowFile or something
that spits out a query or sample that can let the reviewer really see the
submitter's idea of what the input should look like.

Whether those are stored on a Wiki or in the Git repo doesn't really
matter. I think submitting those artifacts will really reduce the burden of
quickly going from "LGTM, JUnits seem to run" to "OK, I see it actually
running as expected."

On Tue, Jan 30, 2018 at 12:38 AM, James Wing  wrote:

> This is a great idea, Mark, thanks for proposing it.  30 days after last
> review comment seems like a good, enforceable standard.
>
> James
>
> On Mon, Jan 29, 2018 at 8:29 AM, Mark Payne  wrote:
>
> > All,
> >
> > We do from time to time go through the backlog of PR's that need to be
> > reviewed and
> > start a "cleansing" process, closing out any old PR's that appear to have
> > stalled out.
> > When we do this, though, we typically will start sending out e-mails
> > asking if there are
> > any stalled PR's that we shouldn't close and start trying to decipher
> > which ones are okay
> > to close out and which ones are not. This puts quite an onus on the
> > committer who is
> > trying to clean this up. It also can result in having a large number of
> > outstanding Pull Requests,
> > which I believe makes the community look bad because it gives the
> > appearance that we are
> > not doing a good job of being responsive to Pull Requests that are
> > submitted.
> >
> > I would like to propose that we set a new "standard" that is: if we have
> > any Pull Request
> > that has been stalled (and by "stalled" I mean a committer has reviewed
> > the PR and did
> > not merge but asked for clarifications or modifications and the
> > contributor has not pushed
> > any new commit or responded to the comments) for at least 30 days, that
> we
> > go ahead
> > and close the Pull Request (after commenting on the PR that it is being
> > closed due to a lack
> > of activity and that the contributor is more than welcome to open a new
> PR
> > if necessary).
> >
> > I feel like this gives contributors enough time to address concerns and
> it
> > is simple enough
> > to create a new Pull Request if the need arises. Alternatively, if the
> > contributor realizes that
> > they need more time, they can simply comment on the PR that they are
> still
> > interested in
> > working on it but just need more time, and the simple act of commenting
> > will mean that the
> > PR is no longer stalled, as defined above.
> >
> > Any thoughts on such a proposal? Any better alternatives that people have
> > in mind?
> >
> > Thanks
> > -Mark
>