Re: Secure NiFi Cluster Unable to Obtain Buckets from Secure NiFi Registry

[Bryan Bende]

Ryan,

Did you happen to enter the registry client in NiFI using the IP
address of the registry?

I'm not totally sure, but based on that message it seems like its
trying to connect to an IP address, but the certificate of the
registry only contains the hostname of the registry.

-Bryan


On Wed, Feb 21, 2018 at 12:52 PM, Ryan H
 wrote:
> Hi All,
>
> I am running into an issue with connecting to a Secure NiFi Registry
> instance from a Secure NiFi cluster. When trying to place a process group
> under version control, I am getting the following error:
>
> Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeException:
> java.security.cert.CertificateException: No subject alternative names
> matching IP address my-secure-registry-ip found
>
> I have added the DN for each of the Nodes in the cluster to the
> authorizers.xml file on the registry in the usersGroupProvider list. I have
> also added the DN of the secure registry to the usersGroupProvider list on
> the secure NiFi cluster nodes.
>
> Any thoughts?
>
> Thanks,
>
> Ryan H

Re: Secure NiFi Cluster Unable to Obtain Buckets from Secure NiFi Registry

[Ryan H]

Hi Andy,

Yes, thanks for the suggestion. Ultimately that is what I want to do for
this specific situation. I just looked at the toolkit and saw that you are
able to add in SAN's. I am going to try that route.

Cheers,

Ryan H.

On Wed, Feb 21, 2018 at 10:38 AM, Andy LoPresto 
wrote:

> Ryan,
>
> In addition to the solution Bryan pointed out, if you want to be able to
> use IP addresses to identify the registry endpoint, you can also add the IP
> address in the Subject Alternative Names list in the certificate and then
> it will be able to verify the certificate.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 21, 2018, at 10:24 AM, Ryan H 
> wrote:
>
> Hi,
>
> Yes, that looks like it is the issue. I think I have run into this problem
> before using IP's instead of hostnames. I have it working now.
>
> Thanks for the quick response!
>
> -Ryan H
>
> On Wed, Feb 21, 2018 at 10:15 AM, Bryan Bende  wrote:
>
>> Ryan,
>>
>> Did you happen to enter the registry client in NiFI using the IP
>> address of the registry?
>>
>> I'm not totally sure, but based on that message it seems like its
>> trying to connect to an IP address, but the certificate of the
>> registry only contains the hostname of the registry.
>>
>> -Bryan
>>
>>
>> On Wed, Feb 21, 2018 at 12:52 PM, Ryan H
>>  wrote:
>> > Hi All,
>> >
>> > I am running into an issue with connecting to a Secure NiFi Registry
>> > instance from a Secure NiFi cluster. When trying to place a process
>> group
>> > under version control, I am getting the following error:
>> >
>> > Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeExce
>> ption:
>> > java.security.cert.CertificateException: No subject alternative names
>> > matching IP address my-secure-registry-ip found
>> >
>> > I have added the DN for each of the Nodes in the cluster to the
>> > authorizers.xml file on the registry in the usersGroupProvider list. I
>> have
>> > also added the DN of the secure registry to the usersGroupProvider list
>> on
>> > the secure NiFi cluster nodes.
>> >
>> > Any thoughts?
>> >
>> > Thanks,
>> >
>> > Ryan H
>>
>
>
>

Re: Secure NiFi Cluster Unable to Obtain Buckets from Secure NiFi Registry

[Andy LoPresto]

Ryan,

In addition to the solution Bryan pointed out, if you want to be able to use IP 
addresses to identify the registry endpoint, you can also add the IP address in 
the Subject Alternative Names list in the certificate and then it will be able 
to verify the certificate.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 21, 2018, at 10:24 AM, Ryan H  
> wrote:
> 
> Hi,
> 
> Yes, that looks like it is the issue. I think I have run into this problem 
> before using IP's instead of hostnames. I have it working now.
> 
> Thanks for the quick response!
> 
> -Ryan H
> 
> On Wed, Feb 21, 2018 at 10:15 AM, Bryan Bende  > wrote:
> Ryan,
> 
> Did you happen to enter the registry client in NiFI using the IP
> address of the registry?
> 
> I'm not totally sure, but based on that message it seems like its
> trying to connect to an IP address, but the certificate of the
> registry only contains the hostname of the registry.
> 
> -Bryan
> 
> 
> On Wed, Feb 21, 2018 at 12:52 PM, Ryan H
>  > wrote:
> > Hi All,
> >
> > I am running into an issue with connecting to a Secure NiFi Registry
> > instance from a Secure NiFi cluster. When trying to place a process group
> > under version control, I am getting the following error:
> >
> > Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeException:
> > java.security.cert.CertificateException: No subject alternative names
> > matching IP address my-secure-registry-ip found
> >
> > I have added the DN for each of the Nodes in the cluster to the
> > authorizers.xml file on the registry in the usersGroupProvider list. I have
> > also added the DN of the secure registry to the usersGroupProvider list on
> > the secure NiFi cluster nodes.
> >
> > Any thoughts?
> >
> > Thanks,
> >
> > Ryan H
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Secure NiFi Cluster Unable to Obtain Buckets from Secure NiFi Registry

[Ryan H]

Hi,

Yes, that looks like it is the issue. I think I have run into this problem
before using IP's instead of hostnames. I have it working now.

Thanks for the quick response!

-Ryan H

On Wed, Feb 21, 2018 at 10:15 AM, Bryan Bende  wrote:

> Ryan,
>
> Did you happen to enter the registry client in NiFI using the IP
> address of the registry?
>
> I'm not totally sure, but based on that message it seems like its
> trying to connect to an IP address, but the certificate of the
> registry only contains the hostname of the registry.
>
> -Bryan
>
>
> On Wed, Feb 21, 2018 at 12:52 PM, Ryan H
>  wrote:
> > Hi All,
> >
> > I am running into an issue with connecting to a Secure NiFi Registry
> > instance from a Secure NiFi cluster. When trying to place a process group
> > under version control, I am getting the following error:
> >
> > Unable to obtain listing of buckets: javax.net.ssl.
> SSLHandshakeException:
> > java.security.cert.CertificateException: No subject alternative names
> > matching IP address my-secure-registry-ip found
> >
> > I have added the DN for each of the Nodes in the cluster to the
> > authorizers.xml file on the registry in the usersGroupProvider list. I
> have
> > also added the DN of the secure registry to the usersGroupProvider list
> on
> > the secure NiFi cluster nodes.
> >
> > Any thoughts?
> >
> > Thanks,
> >
> > Ryan H
>

Secure NiFi Cluster Unable to Obtain Buckets from Secure NiFi Registry

[Ryan H]

Hi All,

I am running into an issue with connecting to a Secure NiFi Registry
instance from a Secure NiFi cluster. When trying to place a process group
under version control, I am getting the following error:

Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeException:
java.security.cert.CertificateException: No subject alternative names
matching IP address my-secure-registry-ip found

I have added the DN for each of the Nodes in the cluster to the
authorizers.xml file on the registry in the usersGroupProvider list. I have
also added the DN of the secure registry to the usersGroupProvider list on
the secure NiFi cluster nodes.

Any thoughts?

Thanks,

Ryan H

Re: minifi secure connection

[Marc]

Arne,
   Yes, I do know what is wrong.  I intended to follow up to this
E-mail in a bit once a second PR was merged. I split the efforts into
two PRs to make reviewing it easier. I posted that PR a few days ago
and review was completed a little while ago and it has been merged.

   The PR resolved a disparity between how HTTP site to site and Raw
socket site to site have to be configured [1]. This would have led to
tremendous confusion among users to configure them differently so I
made it consistent, while also backwards compatible.

   Please try again, and sorry for poorly communicating the fact that
there were two PRs to solve this. In an effort to make reviewing
easier ( and write a better unit test ) I think I dropped the ball in
communicating intent to you.

   Please me me know if you have any additional problems with master.

   [1] https://github.com/apache/nifi-minifi-cpp/pull/265

On Wed, Feb 21, 2018 at 7:02 AM, Arne Oslebo  wrote:
> Hello Marc,
>
> thank you for committing a fix for the Debian issues. Minifi now compiles
> without any warnings but unfortunately I'm still having some problems
> getting things to work properly. My config.yml is now a copy of the example
> in the readme file where you have a GetFile and a RPG. Using unsecured
> communication everything works fine. I then add a SSLContextService and
> reference it from the RPG. The full config.yml is:
>
> Flow Controller:
> id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
> name: MiNiFi Flow
> Processors:
> - name: GetFile
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 1 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list:
>   Properties:
>   Input Directory: /tmp/test
>   Keep Source File: false
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/key.pem
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Connections:
> - name: TransferFilesToRPG
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
>   source name: GetFile
>   source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   source relationship name: success
>   destination id: 8e7979f9-0161-1000-941e-3be83b4479b0
>   max work queue size: 0
>   max work queue data size: 1 MB
>   flowfile expiration: 60 sec
> Remote Processing Groups:
> - name: NiFi Flow
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
>   url: https://***:8443/nifi
>   timeout: 30 secs
>   yield period: 10 sec
>   Input Ports:
>   - id: 8e7979f9-0161-1000-941e-3be83b4479b0
> name: Input
> max concurrent tasks: 1
> Properties:
>Port: 10433
>Host Name: ***
>SSL Context Service: SSLServiceName
>
> From the nifi-user.log i see that minifi connects and authenticates
> properly. The problem is that when I add a file to the /tmp/test directory I
> get the following error from minifi:
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::sitetosite::RawSiteToSiteClient] [error]
> Site2Site Protocol Version Negotiation failed
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] Have 0 peers
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol,
> yielding
>
> In nifi-app.log I get:
> 2018-02-21 12:56:09,654 ERROR [Site-to-Site Worker Thread-0]
> o.a.n.r.io.socket.ssl.SSLSocketChannel
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@21e3bcdc Failed to
> connect due to {}
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at
> sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
> at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:237)
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
> at
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:166)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-21 12:56:09,655 ERROR [Site-to-Site Worker Thread-0]
> o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to accept
> connection from Socket[unconnected] due to javax.net.ssl.SSLException:
> Inbound closed before receiving peer's close_notify: possible truncation
> attack?
>
> I've tried both nifi-1.5.0 and nifi-1.6

FINAL REMINDER: CFP for Apache EU Roadshow Closes 25th February

[Sharan F]

Hello Apache Supporters and Enthusiasts

This is your FINAL reminder that the Call for Papers (CFP) for the 
Apache EU Roadshow is closing soon. Our Apache EU Roadshow will focus on 
Cloud, IoT, Apache Tomcat, Apache Http and will run from 13-14 June 2018 
in Berlin.
Note that the CFP deadline has been extended to *25*^*th* *February *and 
it will be your final opportunity to submit a talk for thisevent.


Please make your submissions at http://apachecon.com/euroadshow18/

Also note that early bird ticket registrations to attend FOSS Backstage 
including the Apache EU Roadshow, have also been extended and will be 
available until 23^rd February. Please register at 
https://foss-backstage.de/tickets


We look forward to seeing you in Berlin!

Thanks
Sharan Foga, VP Apache Community Development

PLEASE NOTE: You are receiving this message because you are subscribed 
to a user@ or dev@ list of one or more Apache Software Foundation projects.

Re: Registry, sdlc and promotion between environments

[Daniel Chaffelson]

Hi Georg,
Likewise to the toolkit CLI coming in PR2477, this functionality is
targeted for NiPyApi 0.8.0 by the end of the month:
https://github.com/Chaffelson/nipyapi/milestone/1

On Wed, Feb 21, 2018 at 12:37 PM Bryan Bende  wrote:

> Georg,
>
> In addition to what Joe and Andrew mentioned, it really comes down to
> how you want to setup your environment...
>
> In a simple scenario, there would be multiple NiFi instances and a
> shared registry instance. In one of the NiFi instances you build a
> flow, start version control, and save v1 to the registry. In the next
> NiFi instance you would import a new Process Group from the registry
> using v1 of the flow you just created. This post describes this
> scenario [1].
>
> In a more complicated scenario, environments may be disconnected and
> you would have a NiFi + Registry in one environment, and another NiFi
> + Registry in another environment. You would then use the toollkit
> Andrew pointed to as a way to export a flow from registry #1 and
> import it to registry #2.
>
> Thanks,
>
> Bryan
>
> [1]
> https://bryanbende.com/development/2018/01/19/apache-nifi-how-do-i-deploy-my-flow
>
>
> On Wed, Feb 21, 2018 at 7:23 AM, Joe Witt  wrote:
> > yes you can absolutely version flows from one instance/env and use them
> in
> > another.
> >
> > backing the registry storage with git is a good next step.
> >
> > On Feb 21, 2018 6:30 AM, "Andrew Grande"  wrote:
> >>
> >> Yes, Georg, there's something coming up to address exactly that, please
> >> take a look at https://github.com/apache/nifi/pull/2477
> >>
> >> Andrew
> >>
> >> On Wed, Feb 21, 2018, 2:45 AM Georg Heiler 
> >> wrote:
> >>>
> >>> Hi
> >>>
> >>> Can I use the new nifi registry to promote / move my nifi flows between
> >>> environments?
> >>>
> >>> To me it currently looks like I can only click around in a single
> >>> environment and then follow up / diff the changes over time via the
> >>> registry.
> >>>
> >>> Also can the registry integrate with git?
> >>>
> >>> Best
> >>> Georg
>

Re: Deploying flows securely over rest api

[Joe Witt]

sean

i mentioned the UI and developer tools so that you could glean details
about the requests/responses and therefore do your automation.

The registry does become a central service for you to publish and consume
versioned flows from any number of nifi instances and clusters, yes.  But
it doesn't aim to replace any VCS at all.

Thanks
Joe

On Wed, Feb 21, 2018 at 9:10 AM, Sean Marciniak  wrote:

> Hi Joe,
>
> I am trying to do in an automated fashion so using the UI doesn't help me
> in this case.
> I already have a working groovy script that will upload these templates
> when NiFi is not secure but can not when it is.
> I am trying to understand what is required in order to make this work for
> secure mode.
>
> I am glad that the host header issue has been addressed in NiFi 1.6 as
> this will help us with securely deploying NiFi into Kubernetes.
>
> I am not fully convinced that nifi registries are the way to go as they
> are an additional service that aims to replace other VCS such as git, svn.
> I am sure it has its purpose but it is not the correct fit for me.
>
> Sean.
>
> On Wed, Feb 21, 2018 at 1:50 PM, Joe Witt  wrote:
>
>> Sean
>>
>> It is certainly possible and you could do this manually via the UI and
>> your browser and use the browsers dev tools to learn more about the
>> requests.  We have the REST API docs but those aren't always very helpful
>> to understand the recipe of taking a series of actions.
>>
>> With NiFi 1.5 host header we made it harder to configure in some cases
>> which we've relaxed and will arrive in 1.6.0 but i'm not aware of it not
>> allowing proper functions and of course it was a change made to address a
>> security concern which since you're running in secure mode I'm guessing
>> you'll find important.
>>
>> With regard to the registry I will say that it was designed to be a far
>> better answer to what templates could never do.  So what you'll work to
>> learn more about now with templates and a workflow to get you closer the
>> registry already does very well.  With 1.6.0 you'll also have access to a
>> nice CLI to use between NiFi and Registry instances for versioned
>> flows/flow management as well.
>>
>> Thanks
>>
>> On Wed, Feb 21, 2018 at 8:42 AM, Jorge Machado  wrote:
>>
>>> Hi Sean,
>>>
>>> Which error are u getting. It the API has the option for it It should
>>> work.
>>> may be this helps: https://github.com/hermannpencole/nifi-config
>>>
>>> Jorge Machado
>>>
>>>
>>>
>>>
>>>
>>> On 21 Feb 2018, at 13:40, Sean Marciniak  wrote:
>>>
>>> Hey Team,
>>>
>>> We are currently trying to deploy flow templates to NiFi while its
>>> running in secure mode over https.
>>> Do we know if this is possible?
>>> Is there any documentation about doing this?
>>>
>>> I am able to deploy the flow templates when it is running in non secure
>>> mode, but when we enforce secure mode, we are unable to do it.
>>>
>>> We are currently stuck with using NiFiv1.4 due host header issues
>>> introduced in NiFi 1.5 and nifi registry is a unneeded risk we don't want
>>> to take.
>>>
>>>
>>> --
>>> 
>>>
>>> Sean Marciniak
>>>
>>> s...@beamery.com
>>>
>>>
>>> www.beamery.com
>>>
>>> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
>>> *
>>>
>>>
>>>
>>
>
>
> --
> 
>
> Sean Marciniak
>
> s...@beamery.com
>
> www.beamery.com
>
> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
> *
>

Re: Deploying flows securely over rest api

[Sean Marciniak]

Jorge,

The error I am currently getting from NiFi is this:

Caught: javax.net.ssl.SSLHandshakeException: Received fatal alert:
bad_certificate
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at
org.apache.http.conn.ssl.SSLSocketFactory.createLayeredSocket(SSLSocketFactory.java:573)
at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:557)
at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
at groovyx.net.http.RESTClient.get(RESTClient.java:119)
at groovyx.net.http.RESTClient$get.call(Unknown Source)
at NiFiDeploy.loadProcessGroups(NiFiDeploy:346)
at NiFiDeploy.loadProcessGroups(NiFiDeploy)
at NiFiDeploy.handleGracefulShutdown(NiFiDeploy:87)
at NiFiDeploy.run(NiFiDeploy:724)

I am currently using the nifi node's truststore.jks to connect to it.

Thanks,

Sean.

On Wed, Feb 21, 2018 at 2:10 PM, Sean Marciniak  wrote:

> Hi Joe,
>
> I am trying to do in an automated fashion so using the UI doesn't help me
> in this case.
> I already have a working groovy script that will upload these templates
> when NiFi is not secure but can not when it is.
> I am trying to understand what is required in order to make this work for
> secure mode.
>
> I am glad that the host header issue has been addressed in NiFi 1.6 as
> this will help us with securely deploying NiFi into Kubernetes.
>
> I am not fully convinced that nifi registries are the way to go as they
> are an additional service that aims to replace other VCS such as git, svn.
> I am sure it has its purpose but it is not the correct fit for me.
>
> Sean.
>
> On Wed, Feb 21, 2018 at 1:50 PM, Joe Witt  wrote:
>
>> Sean
>>
>> It is certainly possible and you could do this manually via the UI and
>> your browser and use the browsers dev tools to learn more about the
>> requests.  We have the REST API docs but those aren't always very helpful
>> to understand the recipe of taking a series of actions.
>>
>> With NiFi 1.5 host header we made it harder to configure in some cases
>> which we've relaxed and will arrive in 1.6.0 but i'm not aware of it not
>> allowing proper functions and of course it was a change made to address a
>> security concern which since you're running in secure mode I'm guessing
>> you'll find important.
>>
>> With regard to the registry I will say that it was designed to be a far
>> better answer to what templates could never do.  So what you'll work to
>> learn more about now with templates and a workflow to get you closer the
>> registry already does very well.  With 1.6.0 you'll also have access to a
>> nice CLI to use between NiFi and Registry instances for versioned
>> flows/flow management as well.
>>
>> Thanks
>>
>> On Wed, Feb 21, 2018 at 8:42 AM, Jorge Machado  wrote:
>>
>>> Hi Sean,
>>>
>>> Which error are u getting. It the API has the option for it It should
>>> work.
>>> may be this helps: https://github.com/hermannpencole/nifi-config
>>>
>>> Jorge Machado
>>>
>>>
>>>
>>>
>>>
>>> On 21 Feb 2018, at 13:40, Sean Marciniak  wrote:
>>>
>>> Hey Team,
>>>
>>> We are currently trying to deploy flow templates to NiFi while its
>>> running in secure mode over https.
>>> Do we know if this is possible?
>>> Is there any documentation about doing this?
>>>
>>> I am able to deploy the flow templates when it is running in non secure
>>> mode, but when we enforce secure mode, we are unable to do it.
>>>
>>> We are currently stuck with using NiFiv1.4 due host header issues
>>> introduced in NiFi 1.5 and nifi registry is a unneeded risk we don't want
>>> to take.
>>>
>>>
>>> --
>>> 
>>>
>>> Sean Marciniak
>>>
>>> s...@beamery.com
>>>
>>>
>>> www.beamery.com
>>>
>>> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
>>> *
>>>
>>>
>>>
>>
>
>
> --
> 
>
> Sean Marciniak
>
> s...@beamery.com
>
> www.beamery.com
>
> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
> *
>



-- 


Sean Marciniak

s...@beamery.com

www.beamery.com

Are y

Re: Deploying flows securely over rest api

[Sean Marciniak]

Hi Joe,

I am trying to do in an automated fashion so using the UI doesn't help me
in this case.
I already have a working groovy script that will upload these templates
when NiFi is not secure but can not when it is.
I am trying to understand what is required in order to make this work for
secure mode.

I am glad that the host header issue has been addressed in NiFi 1.6 as this
will help us with securely deploying NiFi into Kubernetes.

I am not fully convinced that nifi registries are the way to go as they are
an additional service that aims to replace other VCS such as git, svn.
I am sure it has its purpose but it is not the correct fit for me.

Sean.

On Wed, Feb 21, 2018 at 1:50 PM, Joe Witt  wrote:

> Sean
>
> It is certainly possible and you could do this manually via the UI and
> your browser and use the browsers dev tools to learn more about the
> requests.  We have the REST API docs but those aren't always very helpful
> to understand the recipe of taking a series of actions.
>
> With NiFi 1.5 host header we made it harder to configure in some cases
> which we've relaxed and will arrive in 1.6.0 but i'm not aware of it not
> allowing proper functions and of course it was a change made to address a
> security concern which since you're running in secure mode I'm guessing
> you'll find important.
>
> With regard to the registry I will say that it was designed to be a far
> better answer to what templates could never do.  So what you'll work to
> learn more about now with templates and a workflow to get you closer the
> registry already does very well.  With 1.6.0 you'll also have access to a
> nice CLI to use between NiFi and Registry instances for versioned
> flows/flow management as well.
>
> Thanks
>
> On Wed, Feb 21, 2018 at 8:42 AM, Jorge Machado  wrote:
>
>> Hi Sean,
>>
>> Which error are u getting. It the API has the option for it It should
>> work.
>> may be this helps: https://github.com/hermannpencole/nifi-config
>>
>> Jorge Machado
>>
>>
>>
>>
>>
>> On 21 Feb 2018, at 13:40, Sean Marciniak  wrote:
>>
>> Hey Team,
>>
>> We are currently trying to deploy flow templates to NiFi while its
>> running in secure mode over https.
>> Do we know if this is possible?
>> Is there any documentation about doing this?
>>
>> I am able to deploy the flow templates when it is running in non secure
>> mode, but when we enforce secure mode, we are unable to do it.
>>
>> We are currently stuck with using NiFiv1.4 due host header issues
>> introduced in NiFi 1.5 and nifi registry is a unneeded risk we don't want
>> to take.
>>
>>
>> --
>> 
>>
>> Sean Marciniak
>>
>> s...@beamery.com
>>
>>
>> www.beamery.com
>>
>> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
>> *
>>
>>
>>
>


-- 


Sean Marciniak

s...@beamery.com

www.beamery.com

Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
*

Re: Deploying flows securely over rest api

[Joe Witt]

Sean

It is certainly possible and you could do this manually via the UI and your
browser and use the browsers dev tools to learn more about the requests.
We have the REST API docs but those aren't always very helpful to
understand the recipe of taking a series of actions.

With NiFi 1.5 host header we made it harder to configure in some cases
which we've relaxed and will arrive in 1.6.0 but i'm not aware of it not
allowing proper functions and of course it was a change made to address a
security concern which since you're running in secure mode I'm guessing
you'll find important.

With regard to the registry I will say that it was designed to be a far
better answer to what templates could never do.  So what you'll work to
learn more about now with templates and a workflow to get you closer the
registry already does very well.  With 1.6.0 you'll also have access to a
nice CLI to use between NiFi and Registry instances for versioned
flows/flow management as well.

Thanks

On Wed, Feb 21, 2018 at 8:42 AM, Jorge Machado  wrote:

> Hi Sean,
>
> Which error are u getting. It the API has the option for it It should
> work.
> may be this helps: https://github.com/hermannpencole/nifi-config
>
> Jorge Machado
>
>
>
>
>
> On 21 Feb 2018, at 13:40, Sean Marciniak  wrote:
>
> Hey Team,
>
> We are currently trying to deploy flow templates to NiFi while its running
> in secure mode over https.
> Do we know if this is possible?
> Is there any documentation about doing this?
>
> I am able to deploy the flow templates when it is running in non secure
> mode, but when we enforce secure mode, we are unable to do it.
>
> We are currently stuck with using NiFiv1.4 due host header issues
> introduced in NiFi 1.5 and nifi registry is a unneeded risk we don't want
> to take.
>
>
> --
> 
>
> Sean Marciniak
>
> s...@beamery.com
>
>
> www.beamery.com
>
> Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
> *
>
>
>

Re: Deploying flows securely over rest api

[Jorge Machado]

Hi Sean, 

Which error are u getting. It the API has the option for it It should work. 
may be this helps: https://github.com/hermannpencole/nifi-config 


Jorge Machado





> On 21 Feb 2018, at 13:40, Sean Marciniak  wrote:
> 
> Hey Team,
> 
> We are currently trying to deploy flow templates to NiFi while its running in 
> secure mode over https.
> Do we know if this is possible?
> Is there any documentation about doing this?
> 
> I am able to deploy the flow templates when it is running in non secure mode, 
> but when we enforce secure mode, we are unable to do it.
> 
> We are currently stuck with using NiFiv1.4 due host header issues introduced 
> in NiFi 1.5 and nifi registry is a unneeded risk we don't want to take.
> 
> 
> -- 
>     
> Sean Marciniak
> 
> s...@beamery.com 
> 
> 
> www.beamery.com 
> Are you ready for GDPR? GDPR: The Complete Guide for Recruiting Teams 
> 

Deploying flows securely over rest api

[Sean Marciniak]

Hey Team,

We are currently trying to deploy flow templates to NiFi while its running
in secure mode over https.
Do we know if this is possible?
Is there any documentation about doing this?

I am able to deploy the flow templates when it is running in non secure
mode, but when we enforce secure mode, we are unable to do it.

We are currently stuck with using NiFiv1.4 due host header issues
introduced in NiFi 1.5 and nifi registry is a unneeded risk we don't want
to take.


-- 


Sean Marciniak

s...@beamery.com

www.beamery.com

Are you ready for GDPR? *GDPR: The Complete Guide for Recruiting Teams
*

Re: [Data Flow] File content not read completely

[Mark Payne]

Hey Valencia,

I don't believe that PutFile allows you to append to a file, because doing so 
is rife with
problems if you encounter any kind of error (IOException, for example) or if 
NiFi restarts
in between. Instead, you should take a look at MergeContent. You can set the 
"Merge Strategy"
to "Defragment" in order to re-assemble the FlowFiles that were split apart via 
SplitText.

That being said, splitting the data apart, then using ExtractText, and merging 
back together can
be quite expensive. If your data is JSON or CSV, then you should probably look 
into using the Record-Based
Processors (PublishKafkaRecord, QueryRecord/PartitionRecord). This allows you 
to avoid ever splitting
the data apart to begin with and as a result can perform dramatically better.

Thanks
-Mark


On Feb 21, 2018, at 7:27 AM, Valencia Serrao 
mailto:vser...@us.ibm.com>> wrote:


Hi Mark,

Yes! I could get all the required entries with the respective matched and 
unmatched segregated in different folders. Thanks a lot, Mark!!
My next plan is to check and see how to append all the flowfiles with matched 
entries in one file.

Regards,
Valencia

Valencia Serrao---02/16/2018 06:00:47 PM---Hi Mark, Thanks for 
looking into this. I am trying to put in the components you have suggested. I'll

From: Valencia Serrao/Austin/Contr/IBM
To: marka...@hotmail.com
Cc: users@nifi.apache.org
Date: 02/16/2018 06:00 PM
Subject: Re: [Data Flow] File content not read completely




Hi Mark,

Thanks for looking into this. I am trying to put in the components you have 
suggested. I'll update.

Regards,
Valencia


Mark Payne ---02/15/2018 07:09:32 PM---Valencia, The SplitText 
processor does not change the ‘filename’ attribute of the FlowFile. So you w

From: Mark Payne mailto:marka...@hotmail.com>>
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: 02/15/2018 07:09 PM
Subject: Re: [Data Flow] File content not read completely




Valencia,

The SplitText processor does not change the ‘filename’ attribute of the 
FlowFile. So you will end up with multiple FlowFiles having the same name. 
PutFile may well be overwriting the same file many times - or failing to to 
write the files do to filename conflicts. You can resolve this, if it’s your 
problem, by adding an UpdateAttribute to your flow just before PutFile and 
changing the filename to something unique like ${UUID()} or 
${filename}.${nextInt()}

Hope this helps!

-Mark

Sent from my iPhone

On Feb 15, 2018, at 4:59 AM, Valencia Serrao 
mailto:vser...@us.ibm.com>> wrote:

Hi All,

I've started hands-on with Nifi. Basic flows I was able to do without any
issues. But currently I've tried adding more steps to the flow.

Flow intent: Get a local file, split the text on new line, extract text based 
on regex, Put matched/unmatched data on respective kafka topics and
finally write the kafka contents on the local targets set in PutFile.
Current Flow steps: GetFile, SplitText, ExtractText, PutKafka -( 2 of them,one 
for matched and unmatched), and 2 PutFiles components.

The issue I'm facing is that - after the flow execution I see only one entry in 
each of the 2 PutFile targets and rest of the content is not written to them 
even if the criteria is matched. I feel its not looping through the whole file 
or something like that. But I had read that Nifi flow is executed for all 
contents in source files. Maybe I've missed some config somewhere.

It would be really helpful if anyone could help on this issue.

Regards,
Valencia

Re: Registry, sdlc and promotion between environments

[Bryan Bende]

Georg,

In addition to what Joe and Andrew mentioned, it really comes down to
how you want to setup your environment...

In a simple scenario, there would be multiple NiFi instances and a
shared registry instance. In one of the NiFi instances you build a
flow, start version control, and save v1 to the registry. In the next
NiFi instance you would import a new Process Group from the registry
using v1 of the flow you just created. This post describes this
scenario [1].

In a more complicated scenario, environments may be disconnected and
you would have a NiFi + Registry in one environment, and another NiFi
+ Registry in another environment. You would then use the toollkit
Andrew pointed to as a way to export a flow from registry #1 and
import it to registry #2.

Thanks,

Bryan

[1] 
https://bryanbende.com/development/2018/01/19/apache-nifi-how-do-i-deploy-my-flow


On Wed, Feb 21, 2018 at 7:23 AM, Joe Witt  wrote:
> yes you can absolutely version flows from one instance/env and use them in
> another.
>
> backing the registry storage with git is a good next step.
>
> On Feb 21, 2018 6:30 AM, "Andrew Grande"  wrote:
>>
>> Yes, Georg, there's something coming up to address exactly that, please
>> take a look at https://github.com/apache/nifi/pull/2477
>>
>> Andrew
>>
>> On Wed, Feb 21, 2018, 2:45 AM Georg Heiler 
>> wrote:
>>>
>>> Hi
>>>
>>> Can I use the new nifi registry to promote / move my nifi flows between
>>> environments?
>>>
>>> To me it currently looks like I can only click around in a single
>>> environment and then follow up / diff the changes over time via the
>>> registry.
>>>
>>> Also can the registry integrate with git?
>>>
>>> Best
>>> Georg

Re: [Data Flow] File content not read completely

[Valencia Serrao]

Hi Mark,

Yes! I could get all the required entries with the respective matched and
unmatched segregated in different folders. Thanks a lot, Mark!!
My next plan is to check and see how to append all the flowfiles with
matched entries in one file.

Regards,
Valencia



From:   Valencia Serrao/Austin/Contr/IBM
To: marka...@hotmail.com
Cc: users@nifi.apache.org
Date:   02/16/2018 06:00 PM
Subject:Re: [Data Flow] File content not read completely


Hi Mark,

Thanks for looking into this. I am trying to put in the components you have
suggested. I'll update.

Regards,
Valencia




From:   Mark Payne 
To: "users@nifi.apache.org" 
Date:   02/15/2018 07:09 PM
Subject:Re: [Data Flow] File content not read completely



Valencia,

The SplitText processor does not change the ‘filename’ attribute of the
FlowFile. So you will end up with multiple FlowFiles having the same name.
PutFile may well be overwriting the same file many times - or failing to to
write the files do to filename conflicts. You can resolve this, if it’s
your problem, by adding an UpdateAttribute to your flow just before PutFile
and changing the filename to something unique like ${UUID()} or $
{filename}.${nextInt()}

Hope this helps!

-Mark

Sent from my iPhone

On Feb 15, 2018, at 4:59 AM, Valencia Serrao  wrote:



  Hi All,

  I've started hands-on with Nifi. Basic flows I was able to do without
  any
  issues. But currently I've tried adding more steps to the flow.

  Flow intent: Get a local file, split the text on new line, extract
  text based on regex, Put matched/unmatched data on respective kafka
  topics and
  finally write the kafka contents on the local targets set in PutFile.
  Current Flow steps: GetFile, SplitText, ExtractText, PutKafka -( 2 of
  them,one for matched and unmatched), and 2 PutFiles components.

  The issue I'm facing is that - after the flow execution I see only
  one entry in each of the 2 PutFile targets and rest of the content is
  not written to them even if the criteria is matched. I feel its not
  looping through the whole file or something like that. But I had read
  that Nifi flow is executed for all contents in source files. Maybe
  I've missed some config somewhere.

  It would be really helpful if anyone could help on this issue.

  Regards,
  Valencia

Re: Registry, sdlc and promotion between environments

[Joe Witt]

yes you can absolutely version flows from one instance/env and use them in
another.

backing the registry storage with git is a good next step.

On Feb 21, 2018 6:30 AM, "Andrew Grande"  wrote:

> Yes, Georg, there's something coming up to address exactly that, please
> take a look at https://github.com/apache/nifi/pull/2477
>
> Andrew
>
> On Wed, Feb 21, 2018, 2:45 AM Georg Heiler 
> wrote:
>
>> Hi
>>
>> Can I use the new nifi registry to promote / move my nifi flows between
>> environments?
>>
>> To me it currently looks like I can only click around in a single
>> environment and then follow up / diff the changes over time via the
>> registry.
>>
>> Also can the registry integrate with git?
>>
>> Best
>> Georg
>>
>

Re: minifi secure connection

[Arne Oslebo]

Hello Marc,

thank you for committing a fix for the Debian issues. Minifi now
compiles without any warnings but unfortunately I'm still having some
problems getting things to work properly. My config.yml is now a copy of
the example in the readme file where you have a GetFile and a RPG. Using
unsecured communication everything works fine. I then add a
SSLContextService and reference it from the RPG. The full config.yml is:

Flow Controller:
    id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
    name: MiNiFi Flow
Processors:
    - name: GetFile
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 1 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list:
  Properties:
  Input Directory: /tmp/test
  Keep Source File: false
Controller Services:
- name: SSLServiceName
  id: 2438e3c8-015a-1000-79ca-83af40ec1974
  class: SSLContextService
  Properties:
  Client Certificate: /opt/minifi/conf/client.pem
  Private Key: /opt/minifi/conf/key.pem
  CA Certificate: /opt/minifi/conf/nifi-cert.pem
Connections:
    - name: TransferFilesToRPG
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
  source name: GetFile
  source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  source relationship name: success
  destination id: 8e7979f9-0161-1000-941e-3be83b4479b0
  max work queue size: 0
  max work queue data size: 1 MB
  flowfile expiration: 60 sec
Remote Processing Groups:
    - name: NiFi Flow
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
  url: https://***:8443/nifi
  timeout: 30 secs
  yield period: 10 sec
  Input Ports:
  - id: 8e7979f9-0161-1000-941e-3be83b4479b0
    name: Input
    max concurrent tasks: 1
    Properties:
               Port: 10433
   Host Name: ***
   SSL Context Service: SSLServiceName

From the nifi-user.log i see that minifi connects and authenticates
properly. The problem is that when I add a file to the /tmp/test
directory I get the following error from minifi:
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::sitetosite::RawSiteToSiteClient] [error]
Site2Site Protocol Version Negotiation failed
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] Have 0 peers
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no
protocol, yielding

In nifi-app.log I get:
2018-02-21 12:56:09,654 ERROR [Site-to-Site Worker Thread-0]
o.a.n.r.io.socket.ssl.SSLSocketChannel
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@21e3bcdc Failed to
connect due to {}
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
    at
sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:237)
    at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
    at
org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:166)
    at java.lang.Thread.run(Thread.java:748)
2018-02-21 12:56:09,655 ERROR [Site-to-Site Worker Thread-0]
o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to
accept connection from Socket[unconnected] due to
javax.net.ssl.SSLException: Inbound closed before receiving peer's
close_notify: possible truncation attack?

I've tried both nifi-1.5.0 and nifi-1.6.0-SNAPSHOT.

Any suggestions as to what might be wrong?

Best regards,
Arne


On 13/02/2018 18:53, Marc wrote:
> Arne,
>   I took a break from the issue and came back and tried installing a
> different version of openssl on top of the distro. When doing so it
> linked properly and I'm able to send data through a secure Socket. Now
> that I have a solution, I will move this discussion to the ticket. 
>
>   As a result of my testing, I will make updates to the bootstrap
> script and build instructions to instruct users to install libssl1.0
> on Debian Stretch ( and perhaps Raspbian ). Any comments on the ticket
> will be
> appreciated: https://issues.apache.org/jira/browse/MINIFICPP-400
>  . I will have a
> fix once I finish testing across a few platforms. 
>   
>   Thanks,
>   Marc
>
> On Tue, Feb 13, 2018 at 9:55 AM, Marc P.  > wrote:
>
> Arne,
>
> Thanks for the info. I'm running the same environment with the
> same warnings produced -- and segfault -- aso I'll get back to you
> once I've identified the issue.
>
> TL;DR: Crea

Re: Registry, sdlc and promotion between environments

[Andrew Grande]

Yes, Georg, there's something coming up to address exactly that, please
take a look at https://github.com/apache/nifi/pull/2477

Andrew

On Wed, Feb 21, 2018, 2:45 AM Georg Heiler 
wrote:

> Hi
>
> Can I use the new nifi registry to promote / move my nifi flows between
> environments?
>
> To me it currently looks like I can only click around in a single
> environment and then follow up / diff the changes over time via the
> registry.
>
> Also can the registry integrate with git?
>
> Best
> Georg
>