Re: minifi secure connection

[Marc]

Arne,
   Yes, I do know what is wrong.  I intended to follow up to this
E-mail in a bit once a second PR was merged. I split the efforts into
two PRs to make reviewing it easier. I posted that PR a few days ago
and review was completed a little while ago and it has been merged.

   The PR resolved a disparity between how HTTP site to site and Raw
socket site to site have to be configured [1]. This would have led to
tremendous confusion among users to configure them differently so I
made it consistent, while also backwards compatible.

   Please try again, and sorry for poorly communicating the fact that
there were two PRs to solve this. In an effort to make reviewing
easier ( and write a better unit test ) I think I dropped the ball in
communicating intent to you.

   Please me me know if you have any additional problems with master.

   [1] https://github.com/apache/nifi-minifi-cpp/pull/265

On Wed, Feb 21, 2018 at 7:02 AM, Arne Oslebo  wrote:
> Hello Marc,
>
> thank you for committing a fix for the Debian issues. Minifi now compiles
> without any warnings but unfortunately I'm still having some problems
> getting things to work properly. My config.yml is now a copy of the example
> in the readme file where you have a GetFile and a RPG. Using unsecured
> communication everything works fine. I then add a SSLContextService and
> reference it from the RPG. The full config.yml is:
>
> Flow Controller:
> id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
> name: MiNiFi Flow
> Processors:
> - name: GetFile
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 1 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list:
>   Properties:
>   Input Directory: /tmp/test
>   Keep Source File: false
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/key.pem
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Connections:
> - name: TransferFilesToRPG
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
>   source name: GetFile
>   source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   source relationship name: success
>   destination id: 8e7979f9-0161-1000-941e-3be83b4479b0
>   max work queue size: 0
>   max work queue data size: 1 MB
>   flowfile expiration: 60 sec
> Remote Processing Groups:
> - name: NiFi Flow
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
>   url: https://***:8443/nifi
>   timeout: 30 secs
>   yield period: 10 sec
>   Input Ports:
>   - id: 8e7979f9-0161-1000-941e-3be83b4479b0
> name: Input
> max concurrent tasks: 1
> Properties:
>Port: 10433
>Host Name: ***
>SSL Context Service: SSLServiceName
>
> From the nifi-user.log i see that minifi connects and authenticates
> properly. The problem is that when I add a file to the /tmp/test directory I
> get the following error from minifi:
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::sitetosite::RawSiteToSiteClient] [error]
> Site2Site Protocol Version Negotiation failed
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] Have 0 peers
> [2018-02-21 12:56:22.943]
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol,
> yielding
>
> In nifi-app.log I get:
> 2018-02-21 12:56:09,654 ERROR [Site-to-Site Worker Thread-0]
> o.a.n.r.io.socket.ssl.SSLSocketChannel
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@21e3bcdc Failed to
> connect due to {}
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at
> sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
> at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:237)
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
> at
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:166)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-21 12:56:09,655 ERROR [Site-to-Site Worker Thread-0]
> o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to accept
> connection from Socket[unconnected] due to javax.net.ssl.SSLException:
> Inbound closed before receiving peer's close_notify: possible truncation
> attack?
>
> I've tried 

Re: minifi secure connection

[Arne Oslebo]

Hello Marc,

thank you for committing a fix for the Debian issues. Minifi now
compiles without any warnings but unfortunately I'm still having some
problems getting things to work properly. My config.yml is now a copy of
the example in the readme file where you have a GetFile and a RPG. Using
unsecured communication everything works fine. I then add a
SSLContextService and reference it from the RPG. The full config.yml is:

Flow Controller:
    id: 471deef6-2a6e-4a7d-912a-81cc17e3a205
    name: MiNiFi Flow
Processors:
    - name: GetFile
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 1 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list:
  Properties:
  Input Directory: /tmp/test
  Keep Source File: false
Controller Services:
- name: SSLServiceName
  id: 2438e3c8-015a-1000-79ca-83af40ec1974
  class: SSLContextService
  Properties:
  Client Certificate: /opt/minifi/conf/client.pem
  Private Key: /opt/minifi/conf/key.pem
  CA Certificate: /opt/minifi/conf/nifi-cert.pem
Connections:
    - name: TransferFilesToRPG
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a207
  source name: GetFile
  source id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  source relationship name: success
  destination id: 8e7979f9-0161-1000-941e-3be83b4479b0
  max work queue size: 0
  max work queue data size: 1 MB
  flowfile expiration: 60 sec
Remote Processing Groups:
    - name: NiFi Flow
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a208
  url: https://***:8443/nifi
  timeout: 30 secs
  yield period: 10 sec
  Input Ports:
  - id: 8e7979f9-0161-1000-941e-3be83b4479b0
    name: Input
    max concurrent tasks: 1
    Properties:
               Port: 10433
   Host Name: ***
   SSL Context Service: SSLServiceName

From the nifi-user.log i see that minifi connects and authenticates
properly. The problem is that when I add a file to the /tmp/test
directory I get the following error from minifi:
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::sitetosite::RawSiteToSiteClient] [error]
Site2Site Protocol Version Negotiation failed
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] Have 0 peers
[2018-02-21 12:56:22.943]
[org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no
protocol, yielding

In nifi-app.log I get:
2018-02-21 12:56:09,654 ERROR [Site-to-Site Worker Thread-0]
o.a.n.r.io.socket.ssl.SSLSocketChannel
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@21e3bcdc Failed to
connect due to {}
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
    at
sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:237)
    at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
    at
org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:166)
    at java.lang.Thread.run(Thread.java:748)
2018-02-21 12:56:09,655 ERROR [Site-to-Site Worker Thread-0]
o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to
accept connection from Socket[unconnected] due to
javax.net.ssl.SSLException: Inbound closed before receiving peer's
close_notify: possible truncation attack?

I've tried both nifi-1.5.0 and nifi-1.6.0-SNAPSHOT.

Any suggestions as to what might be wrong?

Best regards,
Arne


On 13/02/2018 18:53, Marc wrote:
> Arne,
>   I took a break from the issue and came back and tried installing a
> different version of openssl on top of the distro. When doing so it
> linked properly and I'm able to send data through a secure Socket. Now
> that I have a solution, I will move this discussion to the ticket. 
>
>   As a result of my testing, I will make updates to the bootstrap
> script and build instructions to instruct users to install libssl1.0
> on Debian Stretch ( and perhaps Raspbian ). Any comments on the ticket
> will be
> appreciated: https://issues.apache.org/jira/browse/MINIFICPP-400
>  . I will have a
> fix once I finish testing across a few platforms. 
>   
>   Thanks,
>   Marc
>
> On Tue, Feb 13, 2018 at 9:55 AM, Marc P.  > wrote:
>
> Arne,
>
> Thanks for the info. I'm running the same environment with the
> same warnings produced -- and segfault -- aso I'll get back to you
> once I've identified the 

Re: minifi secure connection

[Marc]

Arne,
  I took a break from the issue and came back and tried installing a
different version of openssl on top of the distro. When doing so it linked
properly and I'm able to send data through a secure Socket. Now that I have
a solution, I will move this discussion to the ticket.

  As a result of my testing, I will make updates to the bootstrap script
and build instructions to instruct users to install libssl1.0 on Debian
Stretch ( and perhaps Raspbian ). Any comments on the ticket will be
appreciated: https://issues.apache.org/jira/browse/MINIFICPP-400 . I will
have a fix once I finish testing across a few platforms.

  Thanks,
  Marc

On Tue, Feb 13, 2018 at 9:55 AM, Marc P.  wrote:

> Arne,
>
> Thanks for the info. I'm running the same environment with the same
> warnings produced -- and segfault -- aso I'll get back to you once I've
> identified the issue.
>
> TL;DR: Created a ticket ( https://issues.apache.org/
> jira/browse/MINIFICPP-400. ) to help insulate users from this more.
>
> More explanation:
>
>
>   In regards to the different versions, there were a number of tickets on
> Debian's bug report logs regarding the curl ABI. An example [1] was solved
> by changing linked versions of libraries.
>
>   The gist is that the SSL_CTX struct changes. I've validated with gdb
> that the struct is being passed in on U16 ( and works ) AND Debian stretch
> ( does not work ). The structures are slightly different. I'm going to dive
> deeper into this. I've created [2] https://issues.apache.org/
> jira/browse/MINIFICPP-400.
>
>[2] has more recent conversation as the issue still exists. This will
> require an soname change hence why they've likely been discussing this for
> two years. I think our job will be to insulate users, so [3] will be our
> efforts to do so. Thanks for identifying this. I'll continue to investigate
> this to address it seamlessly through either our bootstrap script (
> bootstrap.sh in the root ) or within CMAKE. Obviously we don't want to
> alienate Debian users.
>
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844018
> [2] https://issues.apache.org/jira/browse/MINIFICPP-400.
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858398
>
> On Tue, Feb 13, 2018 at 8:28 AM, Arne Oslebo 
> wrote:
>
>> Marc,
>>
>> I'm running it on Debian Stretch. libssl version might be the problem. I
>> see that both libssl1.0.2 and libssl1.1 are installed. I took another look
>> at the build log and found this:
>> /usr/bin/ld: warning: libssl.so.1.0.2, needed by
>> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
>> may conflict with libssl.so.1.1
>> /usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
>> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
>> may conflict with libcrypto.so.1.1
>>
>> I'll see if can remove one of the versions to avoid possible conflict.
>>
>> The full backtrace log is attached and thanks for looking into this,
>>
>> Arne
>>
>>
>>
>> On 13/02/2018 12:55, Marc wrote:
>>
>> Arne,
>>Sorry for your troubles! Can you give me some insight into your distro?
>>
>>I've been unable to replicate the issue on OSX, u16, and arch...but
>> all are running a different version of OpenSSL. What version of OpenSSL are
>> you running and on what distro? I'll be happy to try it to track this down.
>>
>>Additionally, do you have the log file you can pass on? The reason I
>> ask is that the line specified in gdb is a log statement with usage of
>> constructs that are inherent to libstdc++, so they won't cause a memory
>> error. The log file should help identify this and give me insight into what
>> occurred just before the error.
>>
>>Thanks,
>>Marc
>>
>> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo 
>> wrote:
>>
>>> Hello Mark,
>>>
>>> thanks for the update. I tried running master from github but
>>> unfortunately I now get a segmentation fault:
>>>
>>> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
>>> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> (gdb) bt full
>>> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
>>> sl.so.1.1
>>> No symbol table info available.
>>> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
>>> sl.so.1.1
>>> No symbol table info available.
>>> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> No symbol table info available.
>>> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> No symbol table info available.
>>> #4  0x55ef91cb in org::apache::nifi::minifi::con
>>> trollers::SSLContextService::configure_ssl_context
>>> (this=0x569948b0, ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
>>> /extensions/http-curl/../../libminifi/include/controllers/SS
>>> LContextService.h:165
>>> retp = 1
>>> #5  

Re: minifi secure connection

[Marc P.]

Arne,

Thanks for the info. I'm running the same environment with the same
warnings produced -- and segfault -- aso I'll get back to you once I've
identified the issue.

TL;DR: Created a ticket (
https://issues.apache.org/jira/browse/MINIFICPP-400. ) to help insulate
users from this more.

More explanation:


  In regards to the different versions, there were a number of tickets on
Debian's bug report logs regarding the curl ABI. An example [1] was solved
by changing linked versions of libraries.

  The gist is that the SSL_CTX struct changes. I've validated with gdb that
the struct is being passed in on U16 ( and works ) AND Debian stretch (
does not work ). The structures are slightly different. I'm going to dive
deeper into this. I've created [2]
https://issues.apache.org/jira/browse/MINIFICPP-400.

   [2] has more recent conversation as the issue still exists. This will
require an soname change hence why they've likely been discussing this for
two years. I think our job will be to insulate users, so [3] will be our
efforts to do so. Thanks for identifying this. I'll continue to investigate
this to address it seamlessly through either our bootstrap script (
bootstrap.sh in the root ) or within CMAKE. Obviously we don't want to
alienate Debian users.


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844018
[2] https://issues.apache.org/jira/browse/MINIFICPP-400.
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858398

On Tue, Feb 13, 2018 at 8:28 AM, Arne Oslebo  wrote:

> Marc,
>
> I'm running it on Debian Stretch. libssl version might be the problem. I
> see that both libssl1.0.2 and libssl1.1 are installed. I took another look
> at the build log and found this:
> /usr/bin/ld: warning: libssl.so.1.0.2, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may
> conflict with libssl.so.1.1
> /usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may
> conflict with libcrypto.so.1.1
>
> I'll see if can remove one of the versions to avoid possible conflict.
>
> The full backtrace log is attached and thanks for looking into this,
>
> Arne
>
>
>
> On 13/02/2018 12:55, Marc wrote:
>
> Arne,
>Sorry for your troubles! Can you give me some insight into your distro?
>
>I've been unable to replicate the issue on OSX, u16, and arch...but all
> are running a different version of OpenSSL. What version of OpenSSL are you
> running and on what distro? I'll be happy to try it to track this down.
>
>Additionally, do you have the log file you can pass on? The reason I
> ask is that the line specified in gdb is a log statement with usage of
> constructs that are inherent to libstdc++, so they won't cause a memory
> error. The log file should help identify this and give me insight into what
> occurred just before the error.
>
>Thanks,
>Marc
>
> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo 
> wrote:
>
>> Hello Mark,
>>
>> thanks for the update. I tried running master from github but
>> unfortunately I now get a segmentation fault:
>>
>> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
>> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> (gdb) bt full
>> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
>> sl.so.1.1
>> No symbol table info available.
>> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
>> sl.so.1.1
>> No symbol table info available.
>> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> No symbol table info available.
>> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> No symbol table info available.
>> #4  0x55ef91cb in org::apache::nifi::minifi::con
>> trollers::SSLContextService::configure_ssl_context (this=0x569948b0,
>> ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
>> /extensions/http-curl/../../libminifi/include/controllers/SS
>> LContextService.h:165
>> retp = 1
>> #5  0x55ef9be4 in org::apache::nifi::minifi::uti
>> ls::HTTPClient::configure_ssl_context (curl=0x56a149e0,
>> ctx=0x56a28430, param=0x569948b0) at /usr/local/src/nifi-minifi-cpp
>> /extensions/http-curl/client/HTTPClient.h:177
>> ssl_context_service = 0x569948b0
>>
>> Any idea what the problem might be?
>>
>> My full config.yml:
>>
>> Flow Controller:
>>   name: MiNiFi Flow
>> Controller Services:
>> - name: SSLServiceName
>>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>>   class: SSLContextService
>>   Properties:
>>   Client Certificate: /opt/minifi/conf/client.pem
>>   Private Key: /opt/minifi/conf/client.key
>>   Passphrase: secret
>>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
>> Processors:
>> - id: cecb1868-9e5a-3e6c--
>>   name: TailFile
>>   class: 

Re: minifi secure connection

[Arne Oslebo]

Marc,

I'm running it on Debian Stretch. libssl version might be the problem. I
see that both libssl1.0.2 and libssl1.1 are installed. I took another
look at the build log and found this:
/usr/bin/ld: warning: libssl.so.1.0.2, needed by
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
may conflict with libssl.so.1.1
/usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
may conflict with libcrypto.so.1.1

I'll see if can remove one of the versions to avoid possible conflict.

The full backtrace log is attached and thanks for looking into this,

Arne


On 13/02/2018 12:55, Marc wrote:
> Arne,
>    Sorry for your troubles! Can you give me some insight into your distro?
>
>    I've been unable to replicate the issue on OSX, u16, and arch...but
> all are running a different version of OpenSSL. What version of
> OpenSSL are you running and on what distro? I'll be happy to try it to
> track this down. 
>
>    Additionally, do you have the log file you can pass on? The reason
> I ask is that the line specified in gdb is a log statement with usage
> of constructs that are inherent to libstdc++, so they won't cause a
> memory error. The log file should help identify this and give me
> insight into what occurred just before the error.
>
>    Thanks,
>    Marc
>
> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo  > wrote:
>
> Hello Mark,
>
> thanks for the update. I tried running master from github but
> unfortunately I now get a segmentation fault:
>
> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
> 0x7777420a in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> (gdb) bt full
> #0  0x7777420a in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #1  0x77799681 in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #4  0x55ef91cb in
> 
> org::apache::nifi::minifi::controllers::SSLContextService::configure_ssl_context
> (this=0x569948b0, ctx=0x56a28430) at
> 
> /usr/local/src/nifi-minifi-cpp/extensions/http-curl/../../libminifi/include/controllers/SSLContextService.h:165
>     retp = 1
> #5  0x55ef9be4 in
> org::apache::nifi::minifi::utils::HTTPClient::configure_ssl_context
> (curl=0x56a149e0, ctx=0x56a28430, param=0x569948b0) at
> 
> /usr/local/src/nifi-minifi-cpp/extensions/http-curl/client/HTTPClient.h:177
>     ssl_context_service = 0x569948b0
>
> Any idea what the problem might be?
>
> My full config.yml:
>
> Flow Controller:
>   name: MiNiFi Flow
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/client.key
>   Passphrase: secret
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Processors:
> - id: cecb1868-9e5a-3e6c--
>   name: TailFile
>   class: org.apache.nifi.processors.standard.TailFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
>     File Location: Local
>     File to Tail: /tmp/test.log
>     Initial Start Position: Beginning of File
>     Rolling Filename Pattern:
>     tail-base-directory:
>     tail-mode: Single file
>     tailfile-lookup-frequency: 10 minutes
>     tailfile-maximum-age: 24 hours
>     tailfile-recursive-lookup: 'false'
> Connections:
> - id: 76ad4bc4-6557-3e23--
>   name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
>   source id: cecb1868-9e5a-3e6c--
>   source relationship names:
>   - success
>   destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
>   max work queue size: 1
>   max work queue data size: 1 GB
>   flowfile expiration: 0 sec
>   queue prioritizer class: ''
> Remote Processing Groups:
> - id: 3a25e1a3-c1b2-3e78--
>   name: ''
>   url: https://w.x.y.z:8443/nifi
>   comment: ''
>   timeout: 30 sec
>   yield period: 10 sec
>   transport protocol: RAW
>   proxy host: ''
>   proxy port: ''
>   proxy user: ''
> 

Re: minifi secure connection

[Marc]

Arne,
   Sorry for your troubles! Can you give me some insight into your distro?

   I've been unable to replicate the issue on OSX, u16, and arch...but all
are running a different version of OpenSSL. What version of OpenSSL are you
running and on what distro? I'll be happy to try it to track this down.

   Additionally, do you have the log file you can pass on? The reason I ask
is that the line specified in gdb is a log statement with usage of
constructs that are inherent to libstdc++, so they won't cause a memory
error. The log file should help identify this and give me insight into what
occurred just before the error.

   Thanks,
   Marc

On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo  wrote:

> Hello Mark,
>
> thanks for the update. I tried running master from github but
> unfortunately I now get a segmentation fault:
>
> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> (gdb) bt full
> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
> sl.so.1.1
> No symbol table info available.
> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
> sl.so.1.1
> No symbol table info available.
> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #4  0x55ef91cb in org::apache::nifi::minifi::con
> trollers::SSLContextService::configure_ssl_context (this=0x569948b0,
> ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
> /extensions/http-curl/../../libminifi/include/controllers/
> SSLContextService.h:165
> retp = 1
> #5  0x55ef9be4 in org::apache::nifi::minifi::uti
> ls::HTTPClient::configure_ssl_context (curl=0x56a149e0,
> ctx=0x56a28430, param=0x569948b0) at /usr/local/src/nifi-minifi-cpp
> /extensions/http-curl/client/HTTPClient.h:177
> ssl_context_service = 0x569948b0
>
> Any idea what the problem might be?
>
> My full config.yml:
>
> Flow Controller:
>   name: MiNiFi Flow
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/client.key
>   Passphrase: secret
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Processors:
> - id: cecb1868-9e5a-3e6c--
>   name: TailFile
>   class: org.apache.nifi.processors.standard.TailFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> File Location: Local
> File to Tail: /tmp/test.log
> Initial Start Position: Beginning of File
> Rolling Filename Pattern:
> tail-base-directory:
> tail-mode: Single file
> tailfile-lookup-frequency: 10 minutes
> tailfile-maximum-age: 24 hours
> tailfile-recursive-lookup: 'false'
> Connections:
> - id: 76ad4bc4-6557-3e23--
>   name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
>   source id: cecb1868-9e5a-3e6c--
>   source relationship names:
>   - success
>   destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
>   max work queue size: 1
>   max work queue data size: 1 GB
>   flowfile expiration: 0 sec
>   queue prioritizer class: ''
> Remote Processing Groups:
> - id: 3a25e1a3-c1b2-3e78--
>   name: ''
>   url: https://w.x.y.z:8443/nifi
>   comment: ''
>   timeout: 30 sec
>   yield period: 10 sec
>   transport protocol: RAW
>   proxy host: ''
>   proxy port: ''
>   proxy user: ''
>   proxy password: ''
>   local network interface: ''
>   Input Ports:
>   - id: 56ae5abc-0161-1000-aa9e-c340d726e043
> name: Minifi
> comment: ''
> max concurrent tasks: 1
> use compression: false
> Properties:
>   Port: 10443
>   SSL Context Service: SSLServiceName
>   Host Name: w.x.y.z
>   Output Ports: []
> Provenance Reporting:
>
>
>
> On 09/02/2018 20:18, Marc wrote:
>
> Arne,
>   I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263 to
> address these issues.
>
> On Fri, Feb 9, 2018 at 8:38 AM, Marc  wrote:
>
>> Arne,
>>Evidently the HTTPClient relies on an SSL Context Service. Try the
>> following configuration in the config.yml file, where you define the
>> context service and reference it from the RPG. Let me know if that works
>> for you!
>>
>>   Additionally, I think you pointed out an inconsistency where we can
>> improve the configuration and documentation. I've created
>> https://issues.apache.org/jira/browse/MINIFICPP-396 and will take care
>> of that ASAP. Thanks
>>   very much for 

Re: minifi secure connection

[Arne Oslebo]

Hello Mark,

thanks for the update. I tried running master from github but
unfortunately I now get a segmentation fault:

Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
(gdb) bt full
#0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#2  0x7777f2f6 in SSL_CTX_use_certificate () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#4  0x55ef91cb in
org::apache::nifi::minifi::controllers::SSLContextService::configure_ssl_context
(this=0x569948b0, ctx=0x56a28430) at
/usr/local/src/nifi-minifi-cpp/extensions/http-curl/../../libminifi/include/controllers/SSLContextService.h:165
    retp = 1
#5  0x55ef9be4 in
org::apache::nifi::minifi::utils::HTTPClient::configure_ssl_context
(curl=0x56a149e0, ctx=0x56a28430, param=0x569948b0) at
/usr/local/src/nifi-minifi-cpp/extensions/http-curl/client/HTTPClient.h:177
    ssl_context_service = 0x569948b0

Any idea what the problem might be?

My full config.yml:

Flow Controller:
  name: MiNiFi Flow
Controller Services:
- name: SSLServiceName
  id: 2438e3c8-015a-1000-79ca-83af40ec1974
  class: SSLContextService
  Properties:
  Client Certificate: /opt/minifi/conf/client.pem
  Private Key: /opt/minifi/conf/client.key
  Passphrase: secret
  CA Certificate: /opt/minifi/conf/nifi-cert.pem
Processors:
- id: cecb1868-9e5a-3e6c--
  name: TailFile
  class: org.apache.nifi.processors.standard.TailFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    File Location: Local
    File to Tail: /tmp/test.log
    Initial Start Position: Beginning of File
    Rolling Filename Pattern:
    tail-base-directory:
    tail-mode: Single file
    tailfile-lookup-frequency: 10 minutes
    tailfile-maximum-age: 24 hours
    tailfile-recursive-lookup: 'false'
Connections:
- id: 76ad4bc4-6557-3e23--
  name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
  source id: cecb1868-9e5a-3e6c--
  source relationship names:
  - success
  destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
  max work queue size: 1
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- id: 3a25e1a3-c1b2-3e78--
  name: ''
  url: https://w.x.y.z:8443/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  transport protocol: RAW
  proxy host: ''
  proxy port: ''
  proxy user: ''
  proxy password: ''
  local network interface: ''
  Input Ports:
  - id: 56ae5abc-0161-1000-aa9e-c340d726e043
    name: Minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false
    Properties:
  Port: 10443
  SSL Context Service: SSLServiceName
  Host Name: w.x.y.z
  Output Ports: []
Provenance Reporting:


On 09/02/2018 20:18, Marc wrote:
> Arne,
>   I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263
> to address these issues. 
>
> On Fri, Feb 9, 2018 at 8:38 AM, Marc  > wrote:
>
> Arne,
>    Evidently the HTTPClient relies on an SSL Context Service. Try
> the following configuration in the config.yml file, where you
> define the context service and reference it from the RPG. Let me
> know if that works for you!
>
>   Additionally, I think you pointed out an inconsistency where we
> can improve the configuration and documentation. I've
> created https://issues.apache.org/jira/browse/MINIFICPP-396
>  and will
> take care of that ASAP. Thanks 
>   very much for identifying this!
>
> Remote Processing Groups:
>     - name: NiFi Flow
>       id: 2438e3c8-015a-1000-79ca-83af40ec1998
>       url: https://127.0.0.1:8383/nifi
>       timeout: 30 secs
>       yield period: 5 sec
>       Input Ports:
>           - id: 2438e3c8-015a-1000-79ca-83af40ec1999
>             name: fromnifi
>             max concurrent tasks: 1
>             Properties:
>                 Port: 10443
>                 SSL Context Service: SSLMe
>                 Host Name: 127.0.0.1
>       Output Ports:
>           - id: ac82e521-015c-1000-2b21-41279516e19a
>             name: tominifi
>             max concurrent tasks: 2
>             Properties:
>                 Port: 10443
>                 SSL Context Service: SSLMe

Re: minifi secure connection

[Marc]

Arne,
  I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263 to
address these issues.

On Fri, Feb 9, 2018 at 8:38 AM, Marc  wrote:

> Arne,
>Evidently the HTTPClient relies on an SSL Context Service. Try the
> following configuration in the config.yml file, where you define the
> context service and reference it from the RPG. Let me know if that works
> for you!
>
>   Additionally, I think you pointed out an inconsistency where we can
> improve the configuration and documentation. I've created
> https://issues.apache.org/jira/browse/MINIFICPP-396 and will take care of
> that ASAP. Thanks
>   very much for identifying this!
>
> Remote Processing Groups:
> - name: NiFi Flow
>   id: 2438e3c8-015a-1000-79ca-83af40ec1998
>   url: https://127.0.0.1:8383/nifi
>   timeout: 30 secs
>   yield period: 5 sec
>   Input Ports:
>   - id: 2438e3c8-015a-1000-79ca-83af40ec1999
> name: fromnifi
> max concurrent tasks: 1
> Properties:
> Port: 10443
> SSL Context Service: SSLMe
> Host Name: 127.0.0.1
>   Output Ports:
>   - id: ac82e521-015c-1000-2b21-41279516e19a
> name: tominifi
> max concurrent tasks: 2
> Properties:
> Port: 10443
> SSL Context Service: SSLMe
> Host Name: 127.0.0.1
>
> Controller Services:
> - name: SSLMe
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/client.key
>   Passphrase: /opt/minifi/conf/password
>   CA Certificate certificate: /opt/minifi/conf/nifi-cert.pem
>
> On Fri, Feb 9, 2018 at 5:54 AM, Arne Oslebo 
> wrote:
>
>> Hello,
>>
>> I'm trying to set up secure communication between minifi-cpp 0.4.0 and
>> nifi, but unfortunately it fails with the following errors:
>>
>> [org::apache::nifi::minifi::utils::HTTPClient] [error]
>> curl_easy_perform() failed SSL connect error
>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error]
>> ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed
>>
>> I looked quickly at the code and it seems the problem is that HTTPClient
>> never calls configure_secure_connection and therefor never presents a
>> client certificate to nifi.
>>
>> The config.yml file defines a TailFail that send data directly to a
>> remote process group.
>>
>> My  minifi.properties file:
>> nifi.version=0.1.0
>> nifi.flow.configuration.file=/opt/minifi/conf/config.yml
>> nifi.administrative.yield.duration=30 sec
>> nifi.bored.yield.duration=10 millis
>> nifi.provenance.repository.directory.default=/opt/minifi/pro
>> venance_repository
>> nifi.provenance.repository.max.storage.time=1 MIN
>> nifi.provenance.repository.max.storage.size=1 MB
>> nifi.remote.input.secure=true
>> nifi.https.need.ClientAuth=true
>> nifi.https.client.certificate=/opt/minifi/conf/client.pem
>> nifi.https.client.private.key=/opt/minifi/conf/client.key
>> nifi.https.client.pass.phrase=/opt/minifi/conf/password
>> nifi.https.client.ca.certificate=/opt/minifi/conf/nifi-cert.pem
>> controller.socket.host=localhost
>> controller.socket.port=9998
>>
>> Certificates and key are correct and have been verified using curl from
>> the command line. Are there any other things I need to do to get minifi
>> to set up a secure connection? As far as I understand the "Security
>> Properties:" in config.yml is only used by the java version of minifi?
>>
>> Thanks,
>> Arne
>>
>>
>

Re: minifi secure connection

[Marc]

Arne,
   Evidently the HTTPClient relies on an SSL Context Service. Try the
following configuration in the config.yml file, where you define the
context service and reference it from the RPG. Let me know if that works
for you!

  Additionally, I think you pointed out an inconsistency where we can
improve the configuration and documentation. I've created
https://issues.apache.org/jira/browse/MINIFICPP-396 and will take care of
that ASAP. Thanks
  very much for identifying this!

Remote Processing Groups:
- name: NiFi Flow
  id: 2438e3c8-015a-1000-79ca-83af40ec1998
  url: https://127.0.0.1:8383/nifi
  timeout: 30 secs
  yield period: 5 sec
  Input Ports:
  - id: 2438e3c8-015a-1000-79ca-83af40ec1999
name: fromnifi
max concurrent tasks: 1
Properties:
Port: 10443
SSL Context Service: SSLMe
Host Name: 127.0.0.1
  Output Ports:
  - id: ac82e521-015c-1000-2b21-41279516e19a
name: tominifi
max concurrent tasks: 2
Properties:
Port: 10443
SSL Context Service: SSLMe
Host Name: 127.0.0.1

Controller Services:
- name: SSLMe
  id: 2438e3c8-015a-1000-79ca-83af40ec1974
  class: SSLContextService
  Properties:
  Client Certificate: /opt/minifi/conf/client.pem
  Private Key: /opt/minifi/conf/client.key
  Passphrase: /opt/minifi/conf/password
  CA Certificate certificate: /opt/minifi/conf/nifi-cert.pem

On Fri, Feb 9, 2018 at 5:54 AM, Arne Oslebo  wrote:

> Hello,
>
> I'm trying to set up secure communication between minifi-cpp 0.4.0 and
> nifi, but unfortunately it fails with the following errors:
>
> [org::apache::nifi::minifi::utils::HTTPClient] [error]
> curl_easy_perform() failed SSL connect error
> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error]
> ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed
>
> I looked quickly at the code and it seems the problem is that HTTPClient
> never calls configure_secure_connection and therefor never presents a
> client certificate to nifi.
>
> The config.yml file defines a TailFail that send data directly to a
> remote process group.
>
> My  minifi.properties file:
> nifi.version=0.1.0
> nifi.flow.configuration.file=/opt/minifi/conf/config.yml
> nifi.administrative.yield.duration=30 sec
> nifi.bored.yield.duration=10 millis
> nifi.provenance.repository.directory.default=/opt/minifi/
> provenance_repository
> nifi.provenance.repository.max.storage.time=1 MIN
> nifi.provenance.repository.max.storage.size=1 MB
> nifi.remote.input.secure=true
> nifi.https.need.ClientAuth=true
> nifi.https.client.certificate=/opt/minifi/conf/client.pem
> nifi.https.client.private.key=/opt/minifi/conf/client.key
> nifi.https.client.pass.phrase=/opt/minifi/conf/password
> nifi.https.client.ca.certificate=/opt/minifi/conf/nifi-cert.pem
> controller.socket.host=localhost
> controller.socket.port=9998
>
> Certificates and key are correct and have been verified using curl from
> the command line. Are there any other things I need to do to get minifi
> to set up a secure connection? As far as I understand the "Security
> Properties:" in config.yml is only used by the java version of minifi?
>
> Thanks,
> Arne
>
>