Re: how to submit bug found in NIFI code

[尹文才]

Thanks Janosch, I managed to submit a NIFI processor bug after creating an
account of JIRA.

The bug pertains to a new NIFI processor called PutDatabaseRecord and the
link of the issue is as blow, please correct me if anything is wrong since
this is the first time I submit a bug for NIFI:

https://issues.apache.org/jira/browse/NIFI-4228

Regards,
Ben

2017-07-25 19:29 GMT+08:00 Woschitz, Janosch <
janosch.wosch...@thinkbiganalytics.com>:

> Hi Ben,
>
> A submission via JIRA would be the best way to submit a bug. All Apache
> projects using the same JIRA installation but you need to sign up in order
> to use them.
>
> You can create an account via https://issues.apache.org/
> jira/secure/Signup!default.jspa
>
> If this should not be possible you can still share you bug report on the
> dev list but I would highly recommend to file it via JIRA.
>
> Regards,
> Janosch
>
>
>
> On 25.07.17, 11:30, "尹文才"  wrote:
>
> >Hi guys, I wonder if any of you knows the correct way to submit bugs found
> >in current NIFI's code, I checked the JIRA page of NIFI and it seems that
> I
> >need to login to be able to submit an issue. Thanks.
> >
> >Regards,
> >Ben
>

Re: NiFi-4215 stackoverflow exception + patch

[Wes Lawrence]

Joe's right, there's a GitHub PR[1] and patch attached in JIRA[2], but I
did say "patch attached" in my email, and forgot to add it; sorry about
that Andy.

Attached this time, if that's convenient. =)

[1] https://github.com/apache/nifi/pull/2034
[2] https://issues.apache.org/jira/browse/NIFI-4215

--Wes

On Tue, Jul 25, 2017 at 3:22 PM, Joe Witt  wrote:

> He has a nice JIRA and PR up.  Think this is just to help find a reviewer.
>
> Thanks
> Joe
>
> On Tue, Jul 25, 2017 at 3:08 PM, Andy LoPresto 
> wrote:
> > Hi Wes,
> >
> > Thanks for discovering this and bringing it to our attention. The mailing
> > list strips attachments, so your patch is not available here. The best
> > course of action is to submit a pull request on GitHub, or attach the
> patch
> > file to the Jira directly. Let us know if you need any assistance with
> these
> > processes. Thanks again.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jul 25, 2017, at 11:34 AM, Wes Lawrence 
> wrote:
> >
> > Hey all,
> >
> > While building some standard templates we'd like to ship to customers, I
> > found an issue where the AvroSchemaRegistry throws a stack overflow
> > exception on Avro schemas that define a record that composes itself, e.g;
> >
> > {
> >  "namespace": "org.apache.nifi.testing",
> >  "name": "CompositRecord",
> >  "type": "record",
> >  "fields": [
> >{
> >  "name": "id",
> >  "type": "int"
> >},
> >{
> >  "name": "value",
> >  "type": "string"
> >},
> >{
> >  "name": "parent",
> >  "type": [
> >"null",
> >"CompositRecord"
> >  ]
> >}
> >  ]
> > }
> >
> >
> > The patch attached should take care of that use case, looking forward to
> > any feedback, and let me know if anything needs to be improved.
> >
> > --Wes
> >
> >
>

Re: NiFi-4215 stackoverflow exception + patch

[Joe Witt]

He has a nice JIRA and PR up.  Think this is just to help find a reviewer.

Thanks
Joe

On Tue, Jul 25, 2017 at 3:08 PM, Andy LoPresto  wrote:
> Hi Wes,
>
> Thanks for discovering this and bringing it to our attention. The mailing
> list strips attachments, so your patch is not available here. The best
> course of action is to submit a pull request on GitHub, or attach the patch
> file to the Jira directly. Let us know if you need any assistance with these
> processes. Thanks again.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 25, 2017, at 11:34 AM, Wes Lawrence  wrote:
>
> Hey all,
>
> While building some standard templates we'd like to ship to customers, I
> found an issue where the AvroSchemaRegistry throws a stack overflow
> exception on Avro schemas that define a record that composes itself, e.g;
>
> {
>  "namespace": "org.apache.nifi.testing",
>  "name": "CompositRecord",
>  "type": "record",
>  "fields": [
>{
>  "name": "id",
>  "type": "int"
>},
>{
>  "name": "value",
>  "type": "string"
>},
>{
>  "name": "parent",
>  "type": [
>"null",
>"CompositRecord"
>  ]
>}
>  ]
> }
>
>
> The patch attached should take care of that use case, looking forward to
> any feedback, and let me know if anything needs to be improved.
>
> --Wes
>
>

Re: NiFi-4215 stackoverflow exception + patch

[Andy LoPresto]

Hi Wes,

Thanks for discovering this and bringing it to our attention. The mailing list 
strips attachments, so your patch is not available here. The best course of 
action is to submit a pull request on GitHub, or attach the patch file to the 
Jira directly. Let us know if you need any assistance with these processes. 
Thanks again.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 25, 2017, at 11:34 AM, Wes Lawrence  wrote:
> 
> Hey all,
> 
> While building some standard templates we'd like to ship to customers, I
> found an issue where the AvroSchemaRegistry throws a stack overflow
> exception on Avro schemas that define a record that composes itself, e.g;
> 
> {
>  "namespace": "org.apache.nifi.testing",
>  "name": "CompositRecord",
>  "type": "record",
>  "fields": [
>{
>  "name": "id",
>  "type": "int"
>},
>{
>  "name": "value",
>  "type": "string"
>},
>{
>  "name": "parent",
>  "type": [
>"null",
>"CompositRecord"
>  ]
>}
>  ]
> }
> 
> 
> The patch attached should take care of that use case, looking forward to
> any feedback, and let me know if anything needs to be improved.
> 
> --Wes



signature.asc
Description: Message signed with OpenPGP using GPGMail

NiFi-4215 stackoverflow exception + patch

[Wes Lawrence]

Hey all,

While building some standard templates we'd like to ship to customers, I
found an issue where the AvroSchemaRegistry throws a stack overflow
exception on Avro schemas that define a record that composes itself, e.g;

{
  "namespace": "org.apache.nifi.testing",
  "name": "CompositRecord",
  "type": "record",
  "fields": [
{
  "name": "id",
  "type": "int"
},
{
  "name": "value",
  "type": "string"
},
{
  "name": "parent",
  "type": [
"null",
"CompositRecord"
  ]
}
  ]
}


The patch attached should take care of that use case, looking forward to
any feedback, and let me know if anything needs to be improved.

--Wes

Re: why cant nifi perform user authentication over http

[Joe Witt]

YuNing

What can we do to help you setup an HTTPS based environment?  We can
support LDAP-based username and password authentication in that
environment.  We've basically taken a "what is the point" approach to
trying to add authentication/authorization in the HTTP only context so
all is based around HTTPS as the entry point.  From there we've put in
a lot of effort to help you choose the most effective
authentication/authorization model for your case.  There are also some
nice toolkit capabilities that come with the release now too to help
with cert creation.

Thanks
Joe

On Tue, Jul 25, 2017 at 12:54 PM, Andy LoPresto  wrote:
> Modifying NiFi’s source code to provide user authentication and
> authorization over HTTP is highly discouraged. Along with the possibility
> for credential leak that Kevin mentioned, any plaintext HTTP request can be
> intercepted, monitored, and modified before being relayed to the NiFi
> application. This means that any and all actions are susceptible to
> malicious changes, and any entity monitoring the network can perform actions
> under the assumed identity of another user. This would be an incredible
> amount of effort and almost definitely pointless.
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 25, 2017, at 7:09 AM, Kevin Doran  wrote:
>
> Hi YuNing,
>
> In your original post, you mentioned a need for multi-tenant authorization.
> For that use case, I would not recommend transmitting passwords, even
> encrypted/hashed passwords, over unencrypted HTTP, as the authorized
> operations would be still be vulnerable to man-in-the-middle (MITM) attacks
> and replay attacks.
>
> As you mentioned, modifying the NiFi source code to allow authorization over
> HTTP instead of HTTPS would be a significant task, and at the end of the day
> would have the vulnerabilities I described. My advice is that it would be a
> better use of time and effort to configure your NiFi server(s) to use HTTPS.
> The NiFi Toolkit [1] [2] includes TLS utilities to make this easier, and
> there are plenty of folks on this list who can assist you if you have
> questions while setting up HTTPS.
>
> If you truly do not need to worry about security for your use case and do
> not want to use HTTPS, then using HTTP without authorization is an option.
>
> Regards,
> Kevin
>
> [1] https://nifi.apache.org/download.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
>
> On 7/24/17, 23:00, "Sam Feng"  wrote:
>
>Hello Kevin，
>
>   Your answers helps me a lot.  Now i am trying to modify nifi`s
> sourcecode to enable http authentication, because the platform where i am
> using nifi is not that sensitive about security, and we use ldap as
> login-identity-providers whitch password is already encrypted by an unique
> key.
>But i find it difficult to modify it`s sourceCode. there so many
> places that limit login and authentication from http, and i have to edit all
> of it, which will certainly take a lot of time to find them.
>Do you have any idea on how to modify nifi`s code more efficiently,
> or if there are  some other way to get what i want.
>
>As you can see my English is poor, thanks for you patience.
>
>Thanks for your reply.
>Best Regards
>YuNing
>
>
>On 2017-07-21 19:07 (+0800), Kevin Doran  wrote:
>
> Hi,
>
> You are correct, NiFi requires an encrypted connection for user
> authentication. This is because client identity is established in one of two
> ways:
>
> - user name & password, which should not be sent over a non-encrypted
> connection
> - client certificate in a two-way TLS (HTTPS) connection
>
> I hope this answers your question. If HTTPS is suitable for your needs, here
> are some resources to help you get started:
>
> - NiFi System Administration Guide, specifically sections on User
> Authentication [1] and Multi-Tenant Authorization [2]
> - Bryan Bende's blog post on NiFi Authorization and Multi-Tenancy [3]
>
> I hope this helps! If you have any questions you can post back to this
> thread.
>
> Regards,
> Kevin
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization
> [3]
> http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>
>
> On 7/21/17, 02:02, "bel...@163.com"  wrote:
>
>
>Hello， I am a developer from china, i recently want to apply
> multi-tenant authorization on nifi, but find that nifi doesn't support
> authorization over http. can you tell me the reason, and can i enable
> authentication over http by modify it's source code.
>
>Thanks for your early reply.
>Best Regards

Re: why cant nifi perform user authentication over http

[Andy LoPresto]

Modifying NiFi’s source code to provide user authentication and authorization 
over HTTP is highly discouraged. Along with the possibility for credential leak 
that Kevin mentioned, any plaintext HTTP request can be intercepted, monitored, 
and modified before being relayed to the NiFi application. This means that any 
and all actions are susceptible to malicious changes, and any entity monitoring 
the network can perform actions under the assumed identity of another user. 
This would be an incredible amount of effort and almost definitely pointless.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 25, 2017, at 7:09 AM, Kevin Doran  wrote:
> 
> Hi YuNing,
> 
> In your original post, you mentioned a need for multi-tenant authorization. 
> For that use case, I would not recommend transmitting passwords, even 
> encrypted/hashed passwords, over unencrypted HTTP, as the authorized 
> operations would be still be vulnerable to man-in-the-middle (MITM) attacks 
> and replay attacks.
> 
> As you mentioned, modifying the NiFi source code to allow authorization over 
> HTTP instead of HTTPS would be a significant task, and at the end of the day 
> would have the vulnerabilities I described. My advice is that it would be a 
> better use of time and effort to configure your NiFi server(s) to use HTTPS. 
> The NiFi Toolkit [1] [2] includes TLS utilities to make this easier, and 
> there are plenty of folks on this list who can assist you if you have 
> questions while setting up HTTPS.
> 
> If you truly do not need to worry about security for your use case and do not 
> want to use HTTPS, then using HTTP without authorization is an option.
> 
> Regards,
> Kevin
> 
> [1] https://nifi.apache.org/download.html
> [2] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
> 
> On 7/24/17, 23:00, "Sam Feng"  wrote:
> 
>Hello Kevin，
> 
>   Your answers helps me a lot.  Now i am trying to modify nifi`s 
> sourcecode to enable http authentication, because the platform where i am 
> using nifi is not that sensitive about security, and we use ldap as 
> login-identity-providers whitch password is already encrypted by an unique 
> key.
>But i find it difficult to modify it`s sourceCode. there so many 
> places that limit login and authentication from http, and i have to edit all 
> of it, which will certainly take a lot of time to find them.
>Do you have any idea on how to modify nifi`s code more efficiently, or 
> if there are  some other way to get what i want.
> 
>As you can see my English is poor, thanks for you patience.
> 
>Thanks for your reply.
>Best Regards
>YuNing
> 
> 
>On 2017-07-21 19:07 (+0800), Kevin Doran  wrote:
>> Hi,
>> 
>> You are correct, NiFi requires an encrypted connection for user 
>> authentication. This is because client identity is established in one of two 
>> ways:
>> 
>> - user name & password, which should not be sent over a non-encrypted 
>> connection
>> - client certificate in a two-way TLS (HTTPS) connection
>> 
>> I hope this answers your question. If HTTPS is suitable for your needs, here 
>> are some resources to help you get started:
>> 
>> - NiFi System Administration Guide, specifically sections on User 
>> Authentication [1] and Multi-Tenant Authorization [2]
>> - Bryan Bende's blog post on NiFi Authorization and Multi-Tenancy [3]
>> 
>> I hope this helps! If you have any questions you can post back to this 
>> thread.
>> 
>> Regards,
>> Kevin
>> 
>> [1] 
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
>> [2] 
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization
>> [3] 
>> http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>> 
>> 
>> On 7/21/17, 02:02, "bel...@163.com"  wrote:
>> 
>> 
>>Hello， I am a developer from china, i recently want to apply 
>> multi-tenant authorization on nifi, but find that nifi doesn't support 
>> authorization over http. can you tell me the reason, and can i enable 
>> authentication over http by modify it's source code.
>> 
>>Thanks for your early reply.
>>Best Regards
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: nifi processor for IPFIX

[Joe Witt]

Chris,

There are no plans that I am aware of.  We'd need to have build a
ListenUDPRecord processor and we'd need an IPFIXRecordReader.  This
would be pretty slick and quite fast.  I *think* Bryan Bende was
working on a ListenTCPRecord so maybe this could be tied into that.

Thanks

On Tue, Jul 25, 2017 at 9:06 AM, Chris Herssens
 wrote:
> Hello All,
>
> Are there plans to implement an IPFIX collector.
> The processor should listen on an UDP port, parse the content and convert
> it to AVRO or JSON
>
> Regards,
>
> Chris

Re: why cant nifi perform user authentication over http

[Kevin Doran]

Hi YuNing,

In your original post, you mentioned a need for multi-tenant authorization. For 
that use case, I would not recommend transmitting passwords, even 
encrypted/hashed passwords, over unencrypted HTTP, as the authorized operations 
would be still be vulnerable to man-in-the-middle (MITM) attacks and replay 
attacks.

As you mentioned, modifying the NiFi source code to allow authorization over 
HTTP instead of HTTPS would be a significant task, and at the end of the day 
would have the vulnerabilities I described. My advice is that it would be a 
better use of time and effort to configure your NiFi server(s) to use HTTPS. 
The NiFi Toolkit [1] [2] includes TLS utilities to make this easier, and there 
are plenty of folks on this list who can assist you if you have questions while 
setting up HTTPS.

If you truly do not need to worry about security for your use case and do not 
want to use HTTPS, then using HTTP without authorization is an option.

Regards, 
Kevin

[1] https://nifi.apache.org/download.html 
[2] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit
 

On 7/24/17, 23:00, "Sam Feng"  wrote:

Hello Kevin，

   Your answers helps me a lot.  Now i am trying to modify nifi`s 
sourcecode to enable http authentication, because the platform where i am using 
nifi is not that sensitive about security, and we use ldap as 
login-identity-providers whitch password is already encrypted by an unique key. 
But i find it difficult to modify it`s sourceCode. there so many places 
that limit login and authentication from http, and i have to edit all of it, 
which will certainly take a lot of time to find them.  
Do you have any idea on how to modify nifi`s code more efficiently, or 
if there are  some other way to get what i want.
   
As you can see my English is poor, thanks for you patience. 

Thanks for your reply.
Best Regards
YuNing


On 2017-07-21 19:07 (+0800), Kevin Doran  wrote: 
> Hi,
> 
> You are correct, NiFi requires an encrypted connection for user 
authentication. This is because client identity is established in one of two 
ways:
> 
> - user name & password, which should not be sent over a non-encrypted 
connection
> - client certificate in a two-way TLS (HTTPS) connection
> 
> I hope this answers your question. If HTTPS is suitable for your needs, 
here are some resources to help you get started:
> 
> - NiFi System Administration Guide, specifically sections on User 
Authentication [1] and Multi-Tenant Authorization [2]
> - Bryan Bende's blog post on NiFi Authorization and Multi-Tenancy [3]
> 
> I hope this helps! If you have any questions you can post back to this 
thread.
> 
> Regards,
> Kevin
> 
> [1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
 
> [2] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization
 
> [3] 
http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
 
> 
> 
> On 7/21/17, 02:02, "bel...@163.com"  wrote:
> 
> 
> Hello， I am a developer from china, i recently want to apply 
multi-tenant authorization on nifi, but find that nifi doesn't support 
authorization over http. can you tell me the reason, and can i enable 
authentication over http by modify it's source code.
> 
> Thanks for your early reply.
> Best Regards
> 
> 
> 
> 
> 
> 

nifi processor for IPFIX

[Chris Herssens]

Hello All,

Are there plans to implement an IPFIX collector.
The processor should listen on an UDP port, parse the content and convert
it to AVRO or JSON

Regards,

Chris

Re: become dev guide

[Joe Skora]

Ahmed,

To subscribe to the dev list, you send an email to
dev-subscr...@nifi.apache.org.

Instructions to subscribe/unsubscribe to any of the lists as well access
the archives are on the Apache NiFi Mailing Lists
 web page.

Regards,
Joe S

On Tue, Jul 25, 2017 at 5:43 AM, Ahmed SEYE 
wrote:

> ***
> Sonatel:Scan antiviral effectue (3)
> ***
> Hi,
>
> Can i subscribe to nifi.
>
> Best regarads,
>
> Ahmed
>

become dev guide

[Ahmed SEYE]

***
Sonatel:Scan antiviral effectue (3)
***
Hi,

Can i subscribe to nifi.

Best regarads,

Ahmed

Re: how to submit bug found in NIFI code

[Woschitz, Janosch]

Hi Ben,

A submission via JIRA would be the best way to submit a bug. All Apache 
projects using the same JIRA installation but you need to sign up in order to 
use them.

You can create an account via 
https://issues.apache.org/jira/secure/Signup!default.jspa

If this should not be possible you can still share you bug report on the dev 
list but I would highly recommend to file it via JIRA.

Regards,
Janosch



On 25.07.17, 11:30, "尹文才"  wrote:

>Hi guys, I wonder if any of you knows the correct way to submit bugs found
>in current NIFI's code, I checked the JIRA page of NIFI and it seems that I
>need to login to be able to submit an issue. Thanks.
>
>Regards,
>Ben

how to submit bug found in NIFI code

[尹文才]

Hi guys, I wonder if any of you knows the correct way to submit bugs found
in current NIFI's code, I checked the JIRA page of NIFI and it seems that I
need to login to be able to submit an issue. Thanks.

Regards,
Ben