USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Mendez Veiga, Iyan]

Hi,

I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running Raspbian.
I have installed all packages in the requirements (and some optional ones, too):
https://github.com/apache/nifi-minifi-cpp#system-requirements

However, when I execute the bootstrap script I cannot enable the USB Camera 
support (since I want to use GetUSBCamera processor) and the only information I 
get is that "Extension cannot be installed due to version of cmake or other 
software".

I couldn't find any information on the Internet. Any one is facing a similar 
issue?

Regards,
Iyán

Ranger-plugin authorises "anonymous" for /flow

[Meixner, Johannes]

I'm trying to harden my NiFi instance's authorizations and auditing using
Ranger (which is backed by an LDAP instance).

In Ranger I have defined a couple of resources defined to be authorized for
the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.

Turns out that if I add `{USER}` to the resource containing "/flow" I can
read the flow as anonymous user, which is exactly the opposite of what I
want.

Some digging last week lead me to believe that this is due to the way
RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be on the
completely wrong track here.

Is there any way to prevent `anonymous` from doing anything in NiFi,
through Ranger?

Best regards
Johannes Meixner


[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#L185-L188

[Nifi 1.5] PutDatabaseRecord and pgsql integer fields

[françois lacombe]

Hi all,

This is my first mail on this feed, as I've discovering NiFi a few days ago.
I'm working for a company involved in software development for utilities
networks operators

One of my first use case to test NiFi is to load a csv file in an existing
pgsql table, which sounds like a pretty common thing.
I'm currently facing issues with integer fields. My csv file contains
numbers and I use GetFile processor to load and PutDatabaseRecord seems to
quote values with single quotes.

PgSQL doesn't like it at all and raise errors (column is of type numeric
but expression is of type character varying

How can I prevent PutDatabaseRecord process to quote my numeric values?


Thank you in advance for answers, best regards

François Lacombe

Re: [Nifi 1.5] PutDatabaseRecord and pgsql integer fields

[Pierre Villard]

Hi François,

How did you define your schema in the PutDatabaseRecord processor?
If you're using a CSV Reader and infer the schema using the header, then
all the fields will be considered as strings.
You probably want to define the schema by yourself so that you can define
the type of each column of your CSV file.

Pierre


2018-04-09 10:44 GMT+02:00 françois lacombe :

> Hi all,
>
> This is my first mail on this feed, as I've discovering NiFi a few days
> ago.
> I'm working for a company involved in software development for utilities
> networks operators
>
> One of my first use case to test NiFi is to load a csv file in an existing
> pgsql table, which sounds like a pretty common thing.
> I'm currently facing issues with integer fields. My csv file contains
> numbers and I use GetFile processor to load and PutDatabaseRecord seems to
> quote values with single quotes.
>
> PgSQL doesn't like it at all and raise errors (column is of type numeric
> but expression is of type character varying
>
> How can I prevent PutDatabaseRecord process to quote my numeric values?
>
>
> Thank you in advance for answers, best regards
>
> François Lacombe
>
>

Re: [Nifi 1.5] PutDatabaseRecord and pgsql integer fields

[françois lacombe]

Hi Pierre,

Thank you for your input
Schema certainly is a key point here. I used a CSVReader with Use Strings
Fields from Header as access strategy.
Moving access strategy to a text-defined schema solve the issue

On side question: in the schema, what are fields name relative to, csv
header or pgsql table?

All the best

François Lacombe

2018-04-09 10:47 GMT+02:00 Pierre Villard :

> Hi François,
>
> How did you define your schema in the PutDatabaseRecord processor?
> If you're using a CSV Reader and infer the schema using the header, then
> all the fields will be considered as strings.
> You probably want to define the schema by yourself so that you can define
> the type of each column of your CSV file.
>
> Pierre
>
>
> 2018-04-09 10:44 GMT+02:00 françois lacombe 
> :
>
>> Hi all,
>>
>> This is my first mail on this feed, as I've discovering NiFi a few days
>> ago.
>> I'm working for a company involved in software development for utilities
>> networks operators
>>
>> One of my first use case to test NiFi is to load a csv file in an
>> existing pgsql table, which sounds like a pretty common thing.
>> I'm currently facing issues with integer fields. My csv file contains
>> numbers and I use GetFile processor to load and PutDatabaseRecord seems to
>> quote values with single quotes.
>>
>> PgSQL doesn't like it at all and raise errors (column is of type numeric
>> but expression is of type character varying
>>
>> How can I prevent PutDatabaseRecord process to quote my numeric values?
>>
>>
>> Thank you in advance for answers, best regards
>>
>> François Lacombe
>>
>>
>

Re: [Nifi 1.5] PutDatabaseRecord and pgsql integer fields

[Pierre Villard]

If I recall correctly, if you set the property "Ignore CSV Header Column
Names" to true, then it'll take the columns in the same order as defined in
the schema.
Same on the DB side (PutDatabaseRecord), with the property "Translate Field
Names". If false, the field names must match the column names exactly, or
the column will not be updated. If true, I think it'll assume that fields
in the input are in the same order as in the database.

Someone more familiar with PutDatabaseRecord can chime in to correct me :)

Pierre

2018-04-09 11:10 GMT+02:00 françois lacombe :

> Hi Pierre,
>
> Thank you for your input
> Schema certainly is a key point here. I used a CSVReader with Use Strings
> Fields from Header as access strategy.
> Moving access strategy to a text-defined schema solve the issue
>
> On side question: in the schema, what are fields name relative to, csv
> header or pgsql table?
>
> All the best
>
> François Lacombe
>
> 2018-04-09 10:47 GMT+02:00 Pierre Villard :
>
>> Hi François,
>>
>> How did you define your schema in the PutDatabaseRecord processor?
>> If you're using a CSV Reader and infer the schema using the header, then
>> all the fields will be considered as strings.
>> You probably want to define the schema by yourself so that you can define
>> the type of each column of your CSV file.
>>
>> Pierre
>>
>>
>> 2018-04-09 10:44 GMT+02:00 françois lacombe > >:
>>
>>> Hi all,
>>>
>>> This is my first mail on this feed, as I've discovering NiFi a few days
>>> ago.
>>> I'm working for a company involved in software development for utilities
>>> networks operators
>>>
>>> One of my first use case to test NiFi is to load a csv file in an
>>> existing pgsql table, which sounds like a pretty common thing.
>>> I'm currently facing issues with integer fields. My csv file contains
>>> numbers and I use GetFile processor to load and PutDatabaseRecord seems to
>>> quote values with single quotes.
>>>
>>> PgSQL doesn't like it at all and raise errors (column is of type numeric
>>> but expression is of type character varying
>>>
>>> How can I prevent PutDatabaseRecord process to quote my numeric values?
>>>
>>>
>>> Thank you in advance for answers, best regards
>>>
>>> François Lacombe
>>>
>>>
>>
>

Re: [Nifi 1.5] PutDatabaseRecord and pgsql integer fields

[françois lacombe]

Nice Pierre, that's consistent with which I'm experiencing

Currently, this is what i'm using to process csv with header names
different from db table :
Access strategy = Schema text property
Schema text = ...
Treat First Line as Header = True
Ignore CSV Header Column Names = False

In the schema, names should match csv headers, and it'll take columns order
to put in pgsql.
As imported files structure come from external sources, we should adapt our
pgsql structure (with buffer tables eventually) and this is an acceptable
way to achieve for us.

All the best

François Lacombe

2018-04-09 11:30 GMT+02:00 Pierre Villard :

> If I recall correctly, if you set the property "Ignore CSV Header Column
> Names" to true, then it'll take the columns in the same order as defined in
> the schema.
> Same on the DB side (PutDatabaseRecord), with the property "Translate
> Field Names". If false, the field names must match the column names
> exactly, or the column will not be updated. If true, I think it'll assume
> that fields in the input are in the same order as in the database.
>
> Someone more familiar with PutDatabaseRecord can chime in to correct me :)
>
> Pierre
>
> 2018-04-09 11:10 GMT+02:00 françois lacombe 
> :
>
>> Hi Pierre,
>>
>> Thank you for your input
>> Schema certainly is a key point here. I used a CSVReader with Use Strings
>> Fields from Header as access strategy.
>> Moving access strategy to a text-defined schema solve the issue
>>
>> On side question: in the schema, what are fields name relative to, csv
>> header or pgsql table?
>>
>> All the best
>>
>> François Lacombe
>>
>> 2018-04-09 10:47 GMT+02:00 Pierre Villard :
>>
>>> Hi François,
>>>
>>> How did you define your schema in the PutDatabaseRecord processor?
>>> If you're using a CSV Reader and infer the schema using the header, then
>>> all the fields will be considered as strings.
>>> You probably want to define the schema by yourself so that you can
>>> define the type of each column of your CSV file.
>>>
>>> Pierre
>>>
>>>
>>> 2018-04-09 10:44 GMT+02:00 françois lacombe <
>>> francois.laco...@dcbrain.com>:
>>>
 Hi all,

 This is my first mail on this feed, as I've discovering NiFi a few days
 ago.
 I'm working for a company involved in software development for
 utilities networks operators

 One of my first use case to test NiFi is to load a csv file in an
 existing pgsql table, which sounds like a pretty common thing.
 I'm currently facing issues with integer fields. My csv file contains
 numbers and I use GetFile processor to load and PutDatabaseRecord seems to
 quote values with single quotes.

 PgSQL doesn't like it at all and raise errors (column is of type
 numeric but expression is of type character varying

 How can I prevent PutDatabaseRecord process to quote my numeric values?


 Thank you in advance for answers, best regards

 François Lacombe


>>>
>>
>

Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Marc P.]

Iyan,
  That one along with a few others required a bit more manual effort
at the time in bootstrap and thus need to be enabled manually.  Enable
that one manually with cmake -DENABLE_USB_CAMERA=1 ..
  In the next version it'll be added to the bootstrap with 0.5.0. I
was running GetUSBCamera on a Pi last night but I don't specifically
recall if there was much more to take into account. I have a branch
where USB Camera can be enabled on PI. I'll test that on a Pi and
submit a PR if it works.
  Thanks,
  Marc

On Mon, Apr 9, 2018 at 3:46 AM, Mendez Veiga, Iyan
 wrote:
> Hi,
>
> I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running Raspbian.
> I have installed all packages in the requirements (and some optional ones, 
> too):
> https://github.com/apache/nifi-minifi-cpp#system-requirements
>
> However, when I execute the bootstrap script I cannot enable the USB Camera 
> support (since I want to use GetUSBCamera processor) and the only information 
> I get is that "Extension cannot be installed due to version of cmake or other 
> software".
>
> I couldn't find any information on the Internet. Any one is facing a similar 
> issue?
>
> Regards,
> Iyán

Re: Ranger-plugin authorises "anonymous" for /flow

[Bryan Bende]

Hello,

I don't see any issue with the code you linked to. It's saying "if the
ranger policies say the operation is allowed, then return approved".

Is '{USER}' a special syntax in Ranger? or are you using that as a
placeholder in email so you don't have to provide the real user
identity?

I haven't seen that syntax before so just trying to understand what
{USER} and {OWNER} mean here.

-Bryan


On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
 wrote:
> I'm trying to harden my NiFi instance's authorizations and auditing using
> Ranger (which is backed by an LDAP instance).
>
> In Ranger I have defined a couple of resources defined to be authorized for
> the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
>
> Turns out that if I add `{USER}` to the resource containing "/flow" I can
> read the flow as anonymous user, which is exactly the opposite of what I
> want.
>
> Some digging last week lead me to believe that this is due to the way
> RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be on the
> completely wrong track here.
>
> Is there any way to prevent `anonymous` from doing anything in NiFi,
> through Ranger?
>
> Best regards
> Johannes Meixner
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#L185-L188

Re: Ranger-plugin authorises "anonymous" for /flow

[Meixner, Johannes]

Hi Bryan,

It's a placeholder in Ranger for usernames, see
https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable

which is used by certain matcher classes.

On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:

> Hello,
>
> I don't see any issue with the code you linked to. It's saying "if the
> ranger policies say the operation is allowed, then return approved".
>
> Is '{USER}' a special syntax in Ranger? or are you using that as a
> placeholder in email so you don't have to provide the real user
> identity?
>
> I haven't seen that syntax before so just trying to understand what
> {USER} and {OWNER} mean here.
>
> -Bryan
>
>
> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>  wrote:
> > I'm trying to harden my NiFi instance's authorizations and auditing using
> > Ranger (which is backed by an LDAP instance).
> >
> > In Ranger I have defined a couple of resources defined to be authorized
> for
> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
> >
> > Turns out that if I add `{USER}` to the resource containing "/flow" I can
> > read the flow as anonymous user, which is exactly the opposite of what I
> > want.
> >
> > Some digging last week lead me to believe that this is due to the way
> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be on
> the
> > completely wrong track here.
> >
> > Is there any way to prevent `anonymous` from doing anything in NiFi,
> > through Ranger?
> >
> > Best regards
> > Johannes Meixner
> >
> >
> > [1]
> > https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/
> java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#
> L185-L188
>

Re: Ranger-plugin authorises "anonymous" for /flow

[Bryan Bende]

Ah thanks for the info, didn't know that.

So you have a policy in Ranger where the resource is "/flow", the
action is READ, and the users/groups is "{USER}", and then you are
saying an "anonymous" user can retrieve the flow?

I'm assuming that since your Ranger is backed by an LDAP, that you
also have configured NiFi's LDAP Login Identity Provider?

If so, then how are you authenticating to NiFi as an "anonymous" user?

In a secure NiFi you should always have to authenticate as some
identity, the anonymous user is only used in an unsecured NiFi when
there is not authentication/authorization taking place.


On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
 wrote:
> Hi Bryan,
>
> It's a placeholder in Ranger for usernames, see
> https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
>
> which is used by certain matcher classes.
>
> On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
>>
>> Hello,
>>
>> I don't see any issue with the code you linked to. It's saying "if the
>> ranger policies say the operation is allowed, then return approved".
>>
>> Is '{USER}' a special syntax in Ranger? or are you using that as a
>> placeholder in email so you don't have to provide the real user
>> identity?
>>
>> I haven't seen that syntax before so just trying to understand what
>> {USER} and {OWNER} mean here.
>>
>> -Bryan
>>
>>
>> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>>  wrote:
>> > I'm trying to harden my NiFi instance's authorizations and auditing
>> > using
>> > Ranger (which is backed by an LDAP instance).
>> >
>> > In Ranger I have defined a couple of resources defined to be authorized
>> > for
>> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
>> >
>> > Turns out that if I add `{USER}` to the resource containing "/flow" I
>> > can
>> > read the flow as anonymous user, which is exactly the opposite of what I
>> > want.
>> >
>> > Some digging last week lead me to believe that this is due to the way
>> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be on
>> > the
>> > completely wrong track here.
>> >
>> > Is there any way to prevent `anonymous` from doing anything in NiFi,
>> > through Ranger?
>> >
>> > Best regards
>> > Johannes Meixner
>> >
>> >
>> > [1]
>> >
>> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#L185-L188
>
>

Re: Ranger-plugin authorises "anonymous" for /flow

[Meixner, Johannes]

> If so, then how are you authenticating to NiFi as an "anonymous" user?

Good question. NiFi uses its LdapProvider (with Authentication Strategy =
SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
RangerNifiAuthorizer to authorise users.

I'm not quite sure yet what authenticates the anonymous user, but I've
found similar reports in a Hortonworks thread [1] without obvious solution.

[1]
https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html

On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:

> Ah thanks for the info, didn't know that.
>
> So you have a policy in Ranger where the resource is "/flow", the
> action is READ, and the users/groups is "{USER}", and then you are
> saying an "anonymous" user can retrieve the flow?
>
> I'm assuming that since your Ranger is backed by an LDAP, that you
> also have configured NiFi's LDAP Login Identity Provider?
>
> If so, then how are you authenticating to NiFi as an "anonymous" user?
>
> In a secure NiFi you should always have to authenticate as some
> identity, the anonymous user is only used in an unsecured NiFi when
> there is not authentication/authorization taking place.
>
>
> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
>  wrote:
> > Hi Bryan,
> >
> > It's a placeholder in Ranger for usernames, see
> > https://cwiki.apache.org/confluence/display/RANGER/
> Support+for+%24username+variable
> >
> > which is used by certain matcher classes.
> >
> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
> >>
> >> Hello,
> >>
> >> I don't see any issue with the code you linked to. It's saying "if the
> >> ranger policies say the operation is allowed, then return approved".
> >>
> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
> >> placeholder in email so you don't have to provide the real user
> >> identity?
> >>
> >> I haven't seen that syntax before so just trying to understand what
> >> {USER} and {OWNER} mean here.
> >>
> >> -Bryan
> >>
> >>
> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
> >>  wrote:
> >> > I'm trying to harden my NiFi instance's authorizations and auditing
> >> > using
> >> > Ranger (which is backed by an LDAP instance).
> >> >
> >> > In Ranger I have defined a couple of resources defined to be
> authorized
> >> > for
> >> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
> >> >
> >> > Turns out that if I add `{USER}` to the resource containing "/flow" I
> >> > can
> >> > read the flow as anonymous user, which is exactly the opposite of
> what I
> >> > want.
> >> >
> >> > Some digging last week lead me to believe that this is due to the way
> >> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be on
> >> > the
> >> > completely wrong track here.
> >> >
> >> > Is there any way to prevent `anonymous` from doing anything in NiFi,
> >> > through Ranger?
> >> >
> >> > Best regards
> >> > Johannes Meixner
> >> >
> >> >
> >> > [1]
> >> >
> >> > https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/
> java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#
> L185-L188
> >
> >
>

Re: Ranger-plugin authorises "anonymous" for /flow

[Bryan Bende]

Ok, so you end up in the NiFi UI with the identity in the top right
saying "Anonymous user" ? (as show in the screenshots of that hwx
forum)

I think we need to dig into what is happening during authentication...

Can you do a fresh login to NiFi using your LDAP credentials, and then
capture the content from nifi-user.log that occurred during this
login?


On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
 wrote:
>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>
> Good question. NiFi uses its LdapProvider (with Authentication Strategy =
> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
> RangerNifiAuthorizer to authorise users.
>
> I'm not quite sure yet what authenticates the anonymous user, but I've found
> similar reports in a Hortonworks thread [1] without obvious solution.
>
> [1]
> https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html
>
> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
>>
>> Ah thanks for the info, didn't know that.
>>
>> So you have a policy in Ranger where the resource is "/flow", the
>> action is READ, and the users/groups is "{USER}", and then you are
>> saying an "anonymous" user can retrieve the flow?
>>
>> I'm assuming that since your Ranger is backed by an LDAP, that you
>> also have configured NiFi's LDAP Login Identity Provider?
>>
>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>
>> In a secure NiFi you should always have to authenticate as some
>> identity, the anonymous user is only used in an unsecured NiFi when
>> there is not authentication/authorization taking place.
>>
>>
>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
>>  wrote:
>> > Hi Bryan,
>> >
>> > It's a placeholder in Ranger for usernames, see
>> >
>> > https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
>> >
>> > which is used by certain matcher classes.
>> >
>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
>> >>
>> >> Hello,
>> >>
>> >> I don't see any issue with the code you linked to. It's saying "if the
>> >> ranger policies say the operation is allowed, then return approved".
>> >>
>> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
>> >> placeholder in email so you don't have to provide the real user
>> >> identity?
>> >>
>> >> I haven't seen that syntax before so just trying to understand what
>> >> {USER} and {OWNER} mean here.
>> >>
>> >> -Bryan
>> >>
>> >>
>> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>> >>  wrote:
>> >> > I'm trying to harden my NiFi instance's authorizations and auditing
>> >> > using
>> >> > Ranger (which is backed by an LDAP instance).
>> >> >
>> >> > In Ranger I have defined a couple of resources defined to be
>> >> > authorized
>> >> > for
>> >> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
>> >> >
>> >> > Turns out that if I add `{USER}` to the resource containing "/flow" I
>> >> > can
>> >> > read the flow as anonymous user, which is exactly the opposite of
>> >> > what I
>> >> > want.
>> >> >
>> >> > Some digging last week lead me to believe that this is due to the way
>> >> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be
>> >> > on
>> >> > the
>> >> > completely wrong track here.
>> >> >
>> >> > Is there any way to prevent `anonymous` from doing anything in NiFi,
>> >> > through Ranger?
>> >> >
>> >> > Best regards
>> >> > Johannes Meixner
>> >> >
>> >> >
>> >> > [1]
>> >> >
>> >> >
>> >> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#L185-L188
>> >
>> >
>
>

Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

In addition to what Marc said, IIRC, libuvc and libusb are the main
dependencies that need to be available. The libuvc library is included in
thirdparty/ and can be build along with minifi. The libusb-dev package is
needed for the build.

As long as those two libraries are built/available, things should build. You
will need to get identifiers for the USB device (using, e.g. lsusb). Additional
info about this should be in PROCESSORS.md.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJay4P3AAoJEG1+mBKNMpIDNHMH/2dUgrbXKME7b9qVWTwjcRvQ
FbsOf9mgHTAhQHtAGDWDSIfnYNhkoLzuuJeQREitBkInA9Obpw84IIXfkWlIDM3Z
6enByZ4l9+tHBARu7I5/iGmoWk/suC9ohTweOfSGvHeZL0r9+P7llWHF6pIk4h97
8uhVERv5Qw84g31bGIM+h/1JpTFMQqFNjnVrtKYY+uCSOicXKCY3uwY1I88c5iXZ
M3kkyYhBAiXt4oFkdIJp0xWCyJvFmkC9cthyy8f7BCMgV6Z7EgffScybYUgUo81Y
L6yYTXJetqKHNDf4u7wDQBfl8iX3FfPqAwn1r5JhJ0IRj3tBkmyTufziTS8Twss=
=dQ7j
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 9, 2018 8:35 AM, Marc P.  wrote:

> ​​
> 
> Iyan,
> 
> That one along with a few others required a bit more manual effort
> 
> at the time in bootstrap and thus need to be enabled manually. Enable
> 
> that one manually with cmake -DENABLE_USB_CAMERA=1 ..
> 
> In the next version it'll be added to the bootstrap with 0.5.0. I
> 
> was running GetUSBCamera on a Pi last night but I don't specifically
> 
> recall if there was much more to take into account. I have a branch
> 
> where USB Camera can be enabled on PI. I'll test that on a Pi and
> 
> submit a PR if it works.
> 
> Thanks,
> 
> Marc
> 
> On Mon, Apr 9, 2018 at 3:46 AM, Mendez Veiga, Iyan
> 
> iyan.mendezve...@arcelormittal.com wrote:
> 
> > Hi,
> > 
> > I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running 
> > Raspbian.
> > 
> > I have installed all packages in the requirements (and some optional ones, 
> > too):
> > 
> > https://github.com/apache/nifi-minifi-cpp#system-requirements
> > 
> > However, when I execute the bootstrap script I cannot enable the USB Camera 
> > support (since I want to use GetUSBCamera processor) and the only 
> > information I get is that "Extension cannot be installed due to version of 
> > cmake or other software".
> > 
> > I couldn't find any information on the Internet. Any one is facing a 
> > similar issue?
> > 
> > Regards,
> > 
> > Iyán

Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Marc P.]

You may need to install libusb-1.0-0-dev. This is part of the reason
why it was disabled for aptitude distros in 0.4.0; however, with this
added to my bootstrap branch I've successfully installed and run
GetUSBCamera on the RPi and other Ubuntu distros ( 32 and 64 bit too )

On Mon, Apr 9, 2018 at 11:17 AM, Andy Christianson
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Iyán,
>
> In addition to what Marc said, IIRC, libuvc and libusb are the main
> dependencies that need to be available. The libuvc library is included in
> thirdparty/ and can be build along with minifi. The libusb-dev package is
> needed for the build.
>
> As long as those two libraries are built/available, things should build. You
> will need to get identifiers for the USB device (using, e.g. lsusb). 
> Additional
> info about this should be in PROCESSORS.md.
>
> Regards,
>
> Andy I.C.
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQEcBAEBAgAGBQJay4P3AAoJEG1+mBKNMpIDNHMH/2dUgrbXKME7b9qVWTwjcRvQ
> FbsOf9mgHTAhQHtAGDWDSIfnYNhkoLzuuJeQREitBkInA9Obpw84IIXfkWlIDM3Z
> 6enByZ4l9+tHBARu7I5/iGmoWk/suC9ohTweOfSGvHeZL0r9+P7llWHF6pIk4h97
> 8uhVERv5Qw84g31bGIM+h/1JpTFMQqFNjnVrtKYY+uCSOicXKCY3uwY1I88c5iXZ
> M3kkyYhBAiXt4oFkdIJp0xWCyJvFmkC9cthyy8f7BCMgV6Z7EgffScybYUgUo81Y
> L6yYTXJetqKHNDf4u7wDQBfl8iX3FfPqAwn1r5JhJ0IRj3tBkmyTufziTS8Twss=
> =dQ7j
> -END PGP SIGNATURE-
>
>
> Sent from ProtonMail, Swiss-based encrypted email.
>
> ‐‐‐ Original Message ‐‐‐
>
> On April 9, 2018 8:35 AM, Marc P.  wrote:
>
>>
>>
>> Iyan,
>>
>> That one along with a few others required a bit more manual effort
>>
>> at the time in bootstrap and thus need to be enabled manually. Enable
>>
>> that one manually with cmake -DENABLE_USB_CAMERA=1 ..
>>
>> In the next version it'll be added to the bootstrap with 0.5.0. I
>>
>> was running GetUSBCamera on a Pi last night but I don't specifically
>>
>> recall if there was much more to take into account. I have a branch
>>
>> where USB Camera can be enabled on PI. I'll test that on a Pi and
>>
>> submit a PR if it works.
>>
>> Thanks,
>>
>> Marc
>>
>> On Mon, Apr 9, 2018 at 3:46 AM, Mendez Veiga, Iyan
>>
>> iyan.mendezve...@arcelormittal.com wrote:
>>
>> > Hi,
>> >
>> > I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running 
>> > Raspbian.
>> >
>> > I have installed all packages in the requirements (and some optional ones, 
>> > too):
>> >
>> > https://github.com/apache/nifi-minifi-cpp#system-requirements
>> >
>> > However, when I execute the bootstrap script I cannot enable the USB 
>> > Camera support (since I want to use GetUSBCamera processor) and the only 
>> > information I get is that "Extension cannot be installed due to version of 
>> > cmake or other software".
>> >
>> > I couldn't find any information on the Internet. Any one is facing a 
>> > similar issue?
>> >
>> > Regards,
>> >
>> > Iyán
>
>

Re: Ranger-plugin authorises "anonymous" for /flow

[Bryan Bende]

Actually, thinking about this more, I think the issue is that you
currently can't use Ranger's {USER} concept because NiFi always has an
"anonymous" user, and there is no way in Ranger to exclude "anonymous"
from {USER}.

What is happening is something like this...

- You enter the NiFi URL in your browser which loads some of the front-end code
- The front-end code then attempts to access a server side resource
that requires authorization
- This request is happening as the anonymous user, and if it fails
with an unauthorized response, then the front-end knows to present the
login page
- In this case, the request is succeeding because of the policy you
have with {USER}

I think one way to support this might be to bring back the "Allow
Anonymous" flag in the Ranger Authorizer config.

This way if this value is set to false, then before doing anything
else the authorizer would check if the incoming identity is
"anonymous" and reject the request before even checking the actual
policies from Ranger.

On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende  wrote:
> Ok, so you end up in the NiFi UI with the identity in the top right
> saying "Anonymous user" ? (as show in the screenshots of that hwx
> forum)
>
> I think we need to dig into what is happening during authentication...
>
> Can you do a fresh login to NiFi using your LDAP credentials, and then
> capture the content from nifi-user.log that occurred during this
> login?
>
>
> On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
>  wrote:
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>
>> Good question. NiFi uses its LdapProvider (with Authentication Strategy =
>> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
>> RangerNifiAuthorizer to authorise users.
>>
>> I'm not quite sure yet what authenticates the anonymous user, but I've found
>> similar reports in a Hortonworks thread [1] without obvious solution.
>>
>> [1]
>> https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html
>>
>> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
>>>
>>> Ah thanks for the info, didn't know that.
>>>
>>> So you have a policy in Ranger where the resource is "/flow", the
>>> action is READ, and the users/groups is "{USER}", and then you are
>>> saying an "anonymous" user can retrieve the flow?
>>>
>>> I'm assuming that since your Ranger is backed by an LDAP, that you
>>> also have configured NiFi's LDAP Login Identity Provider?
>>>
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>>
>>> In a secure NiFi you should always have to authenticate as some
>>> identity, the anonymous user is only used in an unsecured NiFi when
>>> there is not authentication/authorization taking place.
>>>
>>>
>>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
>>>  wrote:
>>> > Hi Bryan,
>>> >
>>> > It's a placeholder in Ranger for usernames, see
>>> >
>>> > https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
>>> >
>>> > which is used by certain matcher classes.
>>> >
>>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> I don't see any issue with the code you linked to. It's saying "if the
>>> >> ranger policies say the operation is allowed, then return approved".
>>> >>
>>> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
>>> >> placeholder in email so you don't have to provide the real user
>>> >> identity?
>>> >>
>>> >> I haven't seen that syntax before so just trying to understand what
>>> >> {USER} and {OWNER} mean here.
>>> >>
>>> >> -Bryan
>>> >>
>>> >>
>>> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>>> >>  wrote:
>>> >> > I'm trying to harden my NiFi instance's authorizations and auditing
>>> >> > using
>>> >> > Ranger (which is backed by an LDAP instance).
>>> >> >
>>> >> > In Ranger I have defined a couple of resources defined to be
>>> >> > authorized
>>> >> > for
>>> >> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
>>> >> >
>>> >> > Turns out that if I add `{USER}` to the resource containing "/flow" I
>>> >> > can
>>> >> > read the flow as anonymous user, which is exactly the opposite of
>>> >> > what I
>>> >> > want.
>>> >> >
>>> >> > Some digging last week lead me to believe that this is due to the way
>>> >> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could be
>>> >> > on
>>> >> > the
>>> >> > completely wrong track here.
>>> >> >
>>> >> > Is there any way to prevent `anonymous` from doing anything in NiFi,
>>> >> > through Ranger?
>>> >> >
>>> >> > Best regards
>>> >> > Johannes Meixner
>>> >> >
>>> >> >
>>> >> > [1]
>>> >> >
>>> >> >
>>> >> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ranger-bundle/nifi-ranger-plugin/src/main/java/org/apache/nifi/ranger/authorization/RangerNiFiAuthorizer.java#L185-L188
>>> >
>>> >
>>
>>

Re: Ranger-plugin authorises "anonymous" for /flow

[Pierre Villard]

It should also be possible to add a deny policy for anonymous user, no?
Not sure if that would the issue though.

Pierre

2018-04-09 17:32 GMT+02:00 Bryan Bende :

> Actually, thinking about this more, I think the issue is that you
> currently can't use Ranger's {USER} concept because NiFi always has an
> "anonymous" user, and there is no way in Ranger to exclude "anonymous"
> from {USER}.
>
> What is happening is something like this...
>
> - You enter the NiFi URL in your browser which loads some of the front-end
> code
> - The front-end code then attempts to access a server side resource
> that requires authorization
> - This request is happening as the anonymous user, and if it fails
> with an unauthorized response, then the front-end knows to present the
> login page
> - In this case, the request is succeeding because of the policy you
> have with {USER}
>
> I think one way to support this might be to bring back the "Allow
> Anonymous" flag in the Ranger Authorizer config.
>
> This way if this value is set to false, then before doing anything
> else the authorizer would check if the incoming identity is
> "anonymous" and reject the request before even checking the actual
> policies from Ranger.
>
> On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende  wrote:
> > Ok, so you end up in the NiFi UI with the identity in the top right
> > saying "Anonymous user" ? (as show in the screenshots of that hwx
> > forum)
> >
> > I think we need to dig into what is happening during authentication...
> >
> > Can you do a fresh login to NiFi using your LDAP credentials, and then
> > capture the content from nifi-user.log that occurred during this
> > login?
> >
> >
> > On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
> >  wrote:
> >>> If so, then how are you authenticating to NiFi as an "anonymous" user?
> >>
> >> Good question. NiFi uses its LdapProvider (with Authentication Strategy
> =
> >> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
> >> RangerNifiAuthorizer to authorise users.
> >>
> >> I'm not quite sure yet what authenticates the anonymous user, but I've
> found
> >> similar reports in a Hortonworks thread [1] without obvious solution.
> >>
> >> [1]
> >> https://community.hortonworks.com/questions/142667/how-to-
> give-permissions-to-users-to-access-nifi-ui.html
> >>
> >> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
> >>>
> >>> Ah thanks for the info, didn't know that.
> >>>
> >>> So you have a policy in Ranger where the resource is "/flow", the
> >>> action is READ, and the users/groups is "{USER}", and then you are
> >>> saying an "anonymous" user can retrieve the flow?
> >>>
> >>> I'm assuming that since your Ranger is backed by an LDAP, that you
> >>> also have configured NiFi's LDAP Login Identity Provider?
> >>>
> >>> If so, then how are you authenticating to NiFi as an "anonymous" user?
> >>>
> >>> In a secure NiFi you should always have to authenticate as some
> >>> identity, the anonymous user is only used in an unsecured NiFi when
> >>> there is not authentication/authorization taking place.
> >>>
> >>>
> >>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
> >>>  wrote:
> >>> > Hi Bryan,
> >>> >
> >>> > It's a placeholder in Ranger for usernames, see
> >>> >
> >>> > https://cwiki.apache.org/confluence/display/RANGER/
> Support+for+%24username+variable
> >>> >
> >>> > which is used by certain matcher classes.
> >>> >
> >>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende 
> wrote:
> >>> >>
> >>> >> Hello,
> >>> >>
> >>> >> I don't see any issue with the code you linked to. It's saying "if
> the
> >>> >> ranger policies say the operation is allowed, then return approved".
> >>> >>
> >>> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
> >>> >> placeholder in email so you don't have to provide the real user
> >>> >> identity?
> >>> >>
> >>> >> I haven't seen that syntax before so just trying to understand what
> >>> >> {USER} and {OWNER} mean here.
> >>> >>
> >>> >> -Bryan
> >>> >>
> >>> >>
> >>> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
> >>> >>  wrote:
> >>> >> > I'm trying to harden my NiFi instance's authorizations and
> auditing
> >>> >> > using
> >>> >> > Ranger (which is backed by an LDAP instance).
> >>> >> >
> >>> >> > In Ranger I have defined a couple of resources defined to be
> >>> >> > authorized
> >>> >> > for
> >>> >> > the nifi nodes' CNs (from SSL certs), `{USER}` and `{OWNER}`.
> >>> >> >
> >>> >> > Turns out that if I add `{USER}` to the resource containing
> "/flow" I
> >>> >> > can
> >>> >> > read the flow as anonymous user, which is exactly the opposite of
> >>> >> > what I
> >>> >> > want.
> >>> >> >
> >>> >> > Some digging last week lead me to believe that this is due to the
> way
> >>> >> > RangerNiFiAuthorizer.java [1] does authorizations. Note, I could
> be
> >>> >> > on
> >>> >> > the
> >>> >> > completely wrong track here.
> >>> >> >
> >>> >> > Is there any way to prevent `anonymous` from doing anything in
> NiFi,
> >>> >> > t

Re: Ranger-plugin authorises "anonymous" for /flow

[Kevin Doran]

Good catch, Bryan. It does sound like that could explain what is going on in 
these cases.

Regarding  this:

>I think one way to support this might be to bring back the "Allow
>Anonymous" flag in the Ranger Authorizer config.
>
>This way if this value is set to false, then before doing anything
>else the authorizer would check if the incoming identity is
>"anonymous" and reject the request before even checking the actual
>policies from Ranger.

Would this be a good setting to have in general, perhaps for managed authorizer 
or in nifi.properties, so that it could be toggled for any configured 
authorizer?

On 4/9/18, 11:32, "Bryan Bende"  wrote:

Actually, thinking about this more, I think the issue is that you
currently can't use Ranger's {USER} concept because NiFi always has an
"anonymous" user, and there is no way in Ranger to exclude "anonymous"
from {USER}.

What is happening is something like this...

- You enter the NiFi URL in your browser which loads some of the front-end 
code
- The front-end code then attempts to access a server side resource
that requires authorization
- This request is happening as the anonymous user, and if it fails
with an unauthorized response, then the front-end knows to present the
login page
- In this case, the request is succeeding because of the policy you
have with {USER}

I think one way to support this might be to bring back the "Allow
Anonymous" flag in the Ranger Authorizer config.

This way if this value is set to false, then before doing anything
else the authorizer would check if the incoming identity is
"anonymous" and reject the request before even checking the actual
policies from Ranger.

On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende  wrote:
> Ok, so you end up in the NiFi UI with the identity in the top right
> saying "Anonymous user" ? (as show in the screenshots of that hwx
> forum)
>
> I think we need to dig into what is happening during authentication...
>
> Can you do a fresh login to NiFi using your LDAP credentials, and then
> capture the content from nifi-user.log that occurred during this
> login?
>
>
> On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
>  wrote:
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>
>> Good question. NiFi uses its LdapProvider (with Authentication Strategy =
>> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
>> RangerNifiAuthorizer to authorise users.
>>
>> I'm not quite sure yet what authenticates the anonymous user, but I've 
found
>> similar reports in a Hortonworks thread [1] without obvious solution.
>>
>> [1]
>> 
https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html
>>
>> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
>>>
>>> Ah thanks for the info, didn't know that.
>>>
>>> So you have a policy in Ranger where the resource is "/flow", the
>>> action is READ, and the users/groups is "{USER}", and then you are
>>> saying an "anonymous" user can retrieve the flow?
>>>
>>> I'm assuming that since your Ranger is backed by an LDAP, that you
>>> also have configured NiFi's LDAP Login Identity Provider?
>>>
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>>
>>> In a secure NiFi you should always have to authenticate as some
>>> identity, the anonymous user is only used in an unsecured NiFi when
>>> there is not authentication/authorization taking place.
>>>
>>>
>>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
>>>  wrote:
>>> > Hi Bryan,
>>> >
>>> > It's a placeholder in Ranger for usernames, see
>>> >
>>> > 
https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
>>> >
>>> > which is used by certain matcher classes.
>>> >
>>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> I don't see any issue with the code you linked to. It's saying "if 
the
>>> >> ranger policies say the operation is allowed, then return approved".
>>> >>
>>> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
>>> >> placeholder in email so you don't have to provide the real user
>>> >> identity?
>>> >>
>>> >> I haven't seen that syntax before so just trying to understand what
>>> >> {USER} and {OWNER} mean here.
>>> >>
>>> >> -Bryan
>>> >>
>>> >>
>>> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>>> >>  wrote:
>>> >> > I'm trying to harden my NiFi instance's authorizations and auditing
>>> >> > using
>>> >> > Ranger (which is backed by an LDAP instance).
>>> >> >
>>> >> > In Ranger I have defined

Re: Ranger-plugin authorises "anonymous" for /flow

[Bryan Bende]

I don't have a Ranger instance at the moment to test, but the "deny"
policy may work.

Regarding the "Allow Anonymous" approach... since it is a function of
the Authorizer, I think it needs to be in the config in
authorizers.xml. Maybe there is a way to make it applicable to all
authorizers by checking for it in some common code before calling the
underlying authorizer.

The file-based authorizer would never have this problem since it
doesn't provide any wild-carding concepts, so the only way an
anonymous user would get authorized is if you explicitly created a
user called "anonymous" and put that user in the policy for "/flow".


On Mon, Apr 9, 2018 at 11:57 AM, Kevin Doran  wrote:
> Good catch, Bryan. It does sound like that could explain what is going on in 
> these cases.
>
> Regarding  this:
>
>>I think one way to support this might be to bring back the "Allow
>>Anonymous" flag in the Ranger Authorizer config.
>>
>>This way if this value is set to false, then before doing anything
>>else the authorizer would check if the incoming identity is
>>"anonymous" and reject the request before even checking the actual
>>policies from Ranger.
>
> Would this be a good setting to have in general, perhaps for managed 
> authorizer or in nifi.properties, so that it could be toggled for any 
> configured authorizer?
>
> On 4/9/18, 11:32, "Bryan Bende"  wrote:
>
> Actually, thinking about this more, I think the issue is that you
> currently can't use Ranger's {USER} concept because NiFi always has an
> "anonymous" user, and there is no way in Ranger to exclude "anonymous"
> from {USER}.
>
> What is happening is something like this...
>
> - You enter the NiFi URL in your browser which loads some of the 
> front-end code
> - The front-end code then attempts to access a server side resource
> that requires authorization
> - This request is happening as the anonymous user, and if it fails
> with an unauthorized response, then the front-end knows to present the
> login page
> - In this case, the request is succeeding because of the policy you
> have with {USER}
>
> I think one way to support this might be to bring back the "Allow
> Anonymous" flag in the Ranger Authorizer config.
>
> This way if this value is set to false, then before doing anything
> else the authorizer would check if the incoming identity is
> "anonymous" and reject the request before even checking the actual
> policies from Ranger.
>
> On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende  wrote:
> > Ok, so you end up in the NiFi UI with the identity in the top right
> > saying "Anonymous user" ? (as show in the screenshots of that hwx
> > forum)
> >
> > I think we need to dig into what is happening during authentication...
> >
> > Can you do a fresh login to NiFi using your LDAP credentials, and then
> > capture the content from nifi-user.log that occurred during this
> > login?
> >
> >
> > On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
> >  wrote:
> >>> If so, then how are you authenticating to NiFi as an "anonymous" user?
> >>
> >> Good question. NiFi uses its LdapProvider (with Authentication 
> Strategy =
> >> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
> >> RangerNifiAuthorizer to authorise users.
> >>
> >> I'm not quite sure yet what authenticates the anonymous user, but I've 
> found
> >> similar reports in a Hortonworks thread [1] without obvious solution.
> >>
> >> [1]
> >> 
> https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html
> >>
> >> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
> >>>
> >>> Ah thanks for the info, didn't know that.
> >>>
> >>> So you have a policy in Ranger where the resource is "/flow", the
> >>> action is READ, and the users/groups is "{USER}", and then you are
> >>> saying an "anonymous" user can retrieve the flow?
> >>>
> >>> I'm assuming that since your Ranger is backed by an LDAP, that you
> >>> also have configured NiFi's LDAP Login Identity Provider?
> >>>
> >>> If so, then how are you authenticating to NiFi as an "anonymous" user?
> >>>
> >>> In a secure NiFi you should always have to authenticate as some
> >>> identity, the anonymous user is only used in an unsecured NiFi when
> >>> there is not authentication/authorization taking place.
> >>>
> >>>
> >>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
> >>>  wrote:
> >>> > Hi Bryan,
> >>> >
> >>> > It's a placeholder in Ranger for usernames, see
> >>> >
> >>> > 
> https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
> >>> >
> >>> > which is used by certain matcher classes.
> >>> >
> >>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  
> wrote

ListSFTP, FetchSFTP Commands

[Nick Kendra]

Hello,

My organization is researching NiFi to retrieve files from customer
networks via SFTP. Some of our customers need to know the exact SFTP
commands that are issued. I looked through the code found 'ls' used for
getListing() and 'get' used for getInputStream(). Do either of these
processors run other commands, such as creating temporary files on the
remote server?

Thanks for your help,
Nick
-- 
Nick Kendra

Re: ListSFTP, FetchSFTP Commands

[Pierre Villard]

Hello Nick,

Unless the FetchSFTP is configured with a completion strategy 'move' /
'delete', the processors do not need write access.
ls / get / cd should be the only commands I think (and the commands to
conenct to the server obviously).

Pierre

2018-04-09 18:37 GMT+02:00 Nick Kendra :

> Hello,
>
> My organization is researching NiFi to retrieve files from customer
> networks via SFTP. Some of our customers need to know the exact SFTP
> commands that are issued. I looked through the code found 'ls' used for
> getListing() and 'get' used for getInputStream(). Do either of these
> processors run other commands, such as creating temporary files on the
> remote server?
>
> Thanks for your help,
> Nick
> --
> Nick Kendra
>

Integrating nifi with cloud based LDAP JumpCloud

[Scott Howell]

I was wondering if there was anyone on the user group that had successfully 
integrated their NIFI authentication to work with Jumpcloud LDAP. I have 
followed the steps Jumpcloud provides with adding the correct credentials to 
the the NIFI login-identity-providers.xml but I am getting an error of “Unable 
to validate the supplied credentials. Please contact the system administrator.” 
In the UI in my nifi-user.log I am seeing [LDAP: error code 32 - No Such 
Object] when its trying to look up the LDAP user.

Scott

Re: Integrating nifi with cloud based LDAP JumpCloud

[Kevin Doran]

Scott,

I've never implemented NiFi with JumpCloud, but speculating as to what could be 
the cause of your error, it could be the User Search Base/Filter configuration 
values. Can you share the contents of your login-identity-providers.xml 
(removing any sensitive values such as ldap credentials)?

Thanks,
Kevin

On 4/9/18, 14:53, "Scott Howell"  wrote:

I was wondering if there was anyone on the user group that had successfully 
integrated their NIFI authentication to work with Jumpcloud LDAP. I have 
followed the steps Jumpcloud provides with adding the correct credentials to 
the the NIFI login-identity-providers.xml but I am getting an error of “Unable 
to validate the supplied credentials. Please contact the system administrator.” 
In the UI in my nifi-user.log I am seeing [LDAP: error code 32 - No Such 
Object] when its trying to look up the LDAP user.

Scott

Re: Integrating nifi with cloud based LDAP JumpCloud

[Scott Howell]

Yep let me send it over.


ldap-provider
org.apache.nifi.ldap.LdapProvider
ANONYMOUS

uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com










FOLLOW
10 secs
10 secs

ldap://ldap.jumpcloud.com:389
ou=Users,o={redacted},dc=jumpcloud,dc=com
uid={0}

USE_USERNAME
12 hours




> On Apr 9, 2018, at 3:01 PM, Kevin Doran  wrote:
> 
> Scott,
> 
> I've never implemented NiFi with JumpCloud, but speculating as to what could 
> be the cause of your error, it could be the User Search Base/Filter 
> configuration values. Can you share the contents of your 
> login-identity-providers.xml (removing any sensitive values such as ldap 
> credentials)?
> 
> Thanks,
> Kevin
> 
> On 4/9/18, 14:53, "Scott Howell"  wrote:
> 
>I was wondering if there was anyone on the user group that had 
> successfully integrated their NIFI authentication to work with Jumpcloud 
> LDAP. I have followed the steps Jumpcloud provides with adding the correct 
> credentials to the the NIFI login-identity-providers.xml but I am getting an 
> error of “Unable to validate the supplied credentials. Please contact the 
> system administrator.” In the UI in my nifi-user.log I am seeing [LDAP: error 
> code 32 - No Such Object] when its trying to look up the LDAP user.
> 
>Scott
> 
> 

Re: Integrating nifi with cloud based LDAP JumpCloud

[Scott Howell]

That is what is inside of 

> On Apr 9, 2018, at 3:03 PM, Scott Howell  wrote:
> 
> Yep let me send it over.
> 
> 
>ldap-provider
>org.apache.nifi.ldap.LdapProvider
>ANONYMOUS
> 
>uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com
>
>
>
>
>
>
>
>
>
> 
>FOLLOW
>10 secs
>10 secs
> 
>ldap://ldap.jumpcloud.com:389
>ou=Users,o={redacted},dc=jumpcloud,dc=com
>uid={0}
> 
>USE_USERNAME
>12 hours
>
> 
> 
> 
>> On Apr 9, 2018, at 3:01 PM, Kevin Doran  wrote:
>> 
>> Scott,
>> 
>> I've never implemented NiFi with JumpCloud, but speculating as to what could 
>> be the cause of your error, it could be the User Search Base/Filter 
>> configuration values. Can you share the contents of your 
>> login-identity-providers.xml (removing any sensitive values such as ldap 
>> credentials)?
>> 
>> Thanks,
>> Kevin
>> 
>> On 4/9/18, 14:53, "Scott Howell"  wrote:
>> 
>>   I was wondering if there was anyone on the user group that had 
>> successfully integrated their NIFI authentication to work with Jumpcloud 
>> LDAP. I have followed the steps Jumpcloud provides with adding the correct 
>> credentials to the the NIFI login-identity-providers.xml but I am getting an 
>> error of “Unable to validate the supplied credentials. Please contact the 
>> system administrator.” In the UI in my nifi-user.log I am seeing [LDAP: 
>> error code 32 - No Such Object] when its trying to look up the LDAP user.
>> 
>>   Scott
>> 
>> 
> 

Re: Integrating nifi with cloud based LDAP JumpCloud

[Andy LoPresto]

Scott,

One note is that since you are using port 389 (plaintext LDAP), your 
credentials are being transmitted in cleartext unless you are enforcing 
START_TLS, and as there is no truststore populated in your config, it does not 
appear you are doing this.

You should read the Jumpcloud instructions on configuring LDAP-as-a-service 
(including creating an LDAP Binding User Account) using SSL/TLS and there are 
some additional resources on configuring this for LDAP below:

https://support.jumpcloud.com/customer/en/portal/articles/2439911 

https://support.jumpcloud.com/customer/en/portal/articles/2440898-jumpcloud-ldaps-ssl-certificate
 

https://www.digitalocean.com/community/tutorials/how-to-encrypt-openldap-connections-using-starttls
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 9, 2018, at 1:04 PM, Scott Howell  wrote:
> 
> That is what is inside of 
> 
>> On Apr 9, 2018, at 3:03 PM, Scott Howell  wrote:
>> 
>> Yep let me send it over.
>> 
>> 
>>   ldap-provider
>>   org.apache.nifi.ldap.LdapProvider
>>   ANONYMOUS
>> 
>>   uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com
>>   
>>   
>>   
>>   
>>   
>>   
>>   
>>   
>>   
>> 
>>   FOLLOW
>>   10 secs
>>   10 secs
>> 
>>   ldap://ldap.jumpcloud.com:389
>>   ou=Users,o={redacted},dc=jumpcloud,dc=com
>>   uid={0}
>> 
>>   USE_USERNAME
>>   12 hours
>>   
>> 
>> 
>> 
>>> On Apr 9, 2018, at 3:01 PM, Kevin Doran  wrote:
>>> 
>>> Scott,
>>> 
>>> I've never implemented NiFi with JumpCloud, but speculating as to what 
>>> could be the cause of your error, it could be the User Search Base/Filter 
>>> configuration values. Can you share the contents of your 
>>> login-identity-providers.xml (removing any sensitive values such as ldap 
>>> credentials)?
>>> 
>>> Thanks,
>>> Kevin
>>> 
>>> On 4/9/18, 14:53, "Scott Howell"  wrote:
>>> 
>>>  I was wondering if there was anyone on the user group that had 
>>> successfully integrated their NIFI authentication to work with Jumpcloud 
>>> LDAP. I have followed the steps Jumpcloud provides with adding the correct 
>>> credentials to the the NIFI login-identity-providers.xml but I am getting 
>>> an error of “Unable to validate the supplied credentials. Please contact 
>>> the system administrator.” In the UI in my nifi-user.log I am seeing [LDAP: 
>>> error code 32 - No Such Object] when its trying to look up the LDAP user.
>>> 
>>>  Scott
>>> 
>>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Integrating nifi with cloud based LDAP JumpCloud

[Scott Howell]

So I changed two things. 

1. I updated the START_TLS 
with START_TLS this allowed nifi to connect to Jumpcloud.

2. USE_DN from USER_USERNAME and 
everything began to work.



> On Apr 9, 2018, at 3:14 PM, Andy LoPresto  wrote:
> 
> Scott,
> 
> One note is that since you are using port 389 (plaintext LDAP), your 
> credentials are being transmitted in cleartext unless you are enforcing 
> START_TLS, and as there is no truststore populated in your config, it does 
> not appear you are doing this. 
> 
> You should read the Jumpcloud instructions on configuring LDAP-as-a-service 
> (including creating an LDAP Binding User Account) using SSL/TLS and there are 
> some additional resources on configuring this for LDAP below:
> 
> https://support.jumpcloud.com/customer/en/portal/articles/2439911 
> 
> https://support.jumpcloud.com/customer/en/portal/articles/2440898-jumpcloud-ldaps-ssl-certificate
>  
> 
> https://www.digitalocean.com/community/tutorials/how-to-encrypt-openldap-connections-using-starttls
>  
> 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider
>  
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Apr 9, 2018, at 1:04 PM, Scott Howell > > wrote:
>> 
>> That is what is inside of 
>> 
>>> On Apr 9, 2018, at 3:03 PM, Scott Howell >> > wrote:
>>> 
>>> Yep let me send it over.
>>> 
>>> 
>>>   ldap-provider
>>>   org.apache.nifi.ldap.LdapProvider
>>>   ANONYMOUS
>>> 
>>>   uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com
>>>   
>>>   
>>>   
>>>   
>>>   
>>>   
>>>   
>>>   
>>>   
>>> 
>>>   FOLLOW
>>>   10 secs
>>>   10 secs
>>> 
>>>   ldap://ldap.jumpcloud.com:389 
>>> 
>>>   ou=Users,o={redacted},dc=jumpcloud,dc=com
>>>   uid={0}
>>> 
>>>   USE_USERNAME
>>>   12 hours
>>>   
>>> 
>>> 
>>> 
 On Apr 9, 2018, at 3:01 PM, Kevin Doran >>> > wrote:
 
 Scott,
 
 I've never implemented NiFi with JumpCloud, but speculating as to what 
 could be the cause of your error, it could be the User Search Base/Filter 
 configuration values. Can you share the contents of your 
 login-identity-providers.xml (removing any sensitive values such as ldap 
 credentials)?
 
 Thanks,
 Kevin
 
 On 4/9/18, 14:53, "Scott Howell" >>> > wrote:
 
  I was wondering if there was anyone on the user group that had 
 successfully integrated their NIFI authentication to work with Jumpcloud 
 LDAP. I have followed the steps Jumpcloud provides with adding the correct 
 credentials to the the NIFI login-identity-providers.xml but I am getting 
 an error of “Unable to validate the supplied credentials. Please contact 
 the system administrator.” In the UI in my nifi-user.log I am seeing 
 [LDAP: error code 32 - No Such Object] when its trying to look up the LDAP 
 user.
 
  Scott
 
 
>>> 
>> 
> 

Re: NiFi 1.6

[Andy LoPresto]

Just to close the loop on this, Apache NiFi 1.6.0 was released on April 8, 
2018. The download is available here [1].

[1] https://nifi.apache.org/download.html 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 6, 2018, at 2:54 AM, Pierre Villard  
> wrote:
> 
> As Joe said: the vote is ongoing and remains opened 72h.
> Unless there is an issue reported in the next 24h, NiFi 1.6 should be out 
> this WE.
> 
> Pierre
> 
> 2018-04-06 11:47 GMT+02:00 Mohit  >:
> Joe,
> 
> What is the expected date?
> 
> 
> 
> From: Joe Witt mailto:joe.w...@gmail.com>>
> Sent: 06 April 2018 02:20
> To: users@nifi.apache.org 
> Subject: Re: NiFi 1.6
> 
> 
> 
> dan
> 
> 
> 
> It is presently working through the release candidate vote process.  As it 
> stands now it could be out tomorrow.
> 
> 
> 
> Please do help by reviewing the rc if you have time.  If you have questions 
> on how to do it just let us know and we can help.
> 
> 
> 
> thanks
> 
> joe
> 
> 
> 
> On Thu, Apr 5, 2018, 1:46 PM dan young  > wrote:
> 
> any updates on when 1.6 is going to drop?
> 
> 
> 
> dano
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

RE: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Mendez Veiga, Iyan]

Thanks for your feedback Marc and Andy. Knowing that it is possible I will try 
to compile it today. Yesterday it took me some time to compile Bazel and 
Tensorflow (had to use an usb as swap device).

Best regards,
Iyán

-Mensaje original-
De: Marc P. [mailto:marc.par...@gmail.com] 
Enviado el: lunes, 09 de abril de 2018 17:23
Para: users@nifi.apache.org; Andy Christianson 
Asunto: Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?

**This Message originated from a Non-ArcelorMittal source**


You may need to install libusb-1.0-0-dev. This is part of the reason why it was 
disabled for aptitude distros in 0.4.0; however, with this added to my 
bootstrap branch I've successfully installed and run GetUSBCamera on the RPi 
and other Ubuntu distros ( 32 and 64 bit too )

On Mon, Apr 9, 2018 at 11:17 AM, Andy Christianson  
wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Iyán,
>
> In addition to what Marc said, IIRC, libuvc and libusb are the main 
> dependencies that need to be available. The libuvc library is included 
> in thirdparty/ and can be build along with minifi. The libusb-dev 
> package is needed for the build.
>
> As long as those two libraries are built/available, things should 
> build. You will need to get identifiers for the USB device (using, 
> e.g. lsusb). Additional info about this should be in PROCESSORS.md.
>
> Regards,
>
> Andy I.C.
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQEcBAEBAgAGBQJay4P3AAoJEG1+mBKNMpIDNHMH/2dUgrbXKME7b9qVWTwjcRvQ
> FbsOf9mgHTAhQHtAGDWDSIfnYNhkoLzuuJeQREitBkInA9Obpw84IIXfkWlIDM3Z
> 6enByZ4l9+tHBARu7I5/iGmoWk/suC9ohTweOfSGvHeZL0r9+P7llWHF6pIk4h97
> 8uhVERv5Qw84g31bGIM+h/1JpTFMQqFNjnVrtKYY+uCSOicXKCY3uwY1I88c5iXZ
> M3kkyYhBAiXt4oFkdIJp0xWCyJvFmkC9cthyy8f7BCMgV6Z7EgffScybYUgUo81Y
> L6yYTXJetqKHNDf4u7wDQBfl8iX3FfPqAwn1r5JhJ0IRj3tBkmyTufziTS8Twss=
> =dQ7j
> -END PGP SIGNATURE-
>
>
> Sent from ProtonMail, Swiss-based encrypted email.
>
> ‐‐‐ Original Message ‐‐‐
>
> On April 9, 2018 8:35 AM, Marc P.  wrote:
>
>>
>>
>> Iyan,
>>
>> That one along with a few others required a bit more manual effort
>>
>> at the time in bootstrap and thus need to be enabled manually. Enable
>>
>> that one manually with cmake -DENABLE_USB_CAMERA=1 ..
>>
>> In the next version it'll be added to the bootstrap with 0.5.0. I
>>
>> was running GetUSBCamera on a Pi last night but I don't specifically
>>
>> recall if there was much more to take into account. I have a branch
>>
>> where USB Camera can be enabled on PI. I'll test that on a Pi and
>>
>> submit a PR if it works.
>>
>> Thanks,
>>
>> Marc
>>
>> On Mon, Apr 9, 2018 at 3:46 AM, Mendez Veiga, Iyan
>>
>> iyan.mendezve...@arcelormittal.com wrote:
>>
>> > Hi,
>> >
>> > I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running 
>> > Raspbian.
>> >
>> > I have installed all packages in the requirements (and some optional ones, 
>> > too):
>> >
>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apa
>> > che_nifi-2Dminifi-2Dcpp-23system-2Drequirements&d=DwIFaQ&c=y5LGzd1h
>> > T50ruE_IlUH7x8VGgWz9W0tFVWT6rSvPUKA&r=OqJOXvZGIFafgLt7Uocjlzsiwprj2
>> > _HJJ0p7HG9IsupasVjguMhbhoUfzdA5V_Qn&m=wPIoEah-YRSi6LhxgnSby8YVO15gZ
>> > BLprUh9HXzl9j0&s=pQzDSBhDQlp-WfrpF3avtFilpTOpeB0M-DKJLWhsJgg&e=
>> >
>> > However, when I execute the bootstrap script I cannot enable the USB 
>> > Camera support (since I want to use GetUSBCamera processor) and the only 
>> > information I get is that "Extension cannot be installed due to version of 
>> > cmake or other software".
>> >
>> > I couldn't find any information on the Internet. Any one is facing a 
>> > similar issue?
>> >
>> > Regards,
>> >
>> > Iyán
>
>