Arne, I took a break from the issue and came back and tried installing a different version of openssl on top of the distro. When doing so it linked properly and I'm able to send data through a secure Socket. Now that I have a solution, I will move this discussion to the ticket.
As a result of my testing, I will make updates to the bootstrap script and build instructions to instruct users to install libssl1.0 on Debian Stretch ( and perhaps Raspbian ). Any comments on the ticket will be appreciated: https://issues.apache.org/jira/browse/MINIFICPP-400 . I will have a fix once I finish testing across a few platforms. Thanks, Marc On Tue, Feb 13, 2018 at 9:55 AM, Marc P. <[email protected]> wrote: > Arne, > > Thanks for the info. I'm running the same environment with the same > warnings produced -- and segfault -- aso I'll get back to you once I've > identified the issue. > > TL;DR: Created a ticket ( https://issues.apache.org/ > jira/browse/MINIFICPP-400. ) to help insulate users from this more. > > More explanation: > > > In regards to the different versions, there were a number of tickets on > Debian's bug report logs regarding the curl ABI. An example [1] was solved > by changing linked versions of libraries. > > The gist is that the SSL_CTX struct changes. I've validated with gdb > that the struct is being passed in on U16 ( and works ) AND Debian stretch > ( does not work ). The structures are slightly different. I'm going to dive > deeper into this. I've created [2] https://issues.apache.org/ > jira/browse/MINIFICPP-400. > > [2] has more recent conversation as the issue still exists. This will > require an soname change hence why they've likely been discussing this for > two years. I think our job will be to insulate users, so [3] will be our > efforts to do so. Thanks for identifying this. I'll continue to investigate > this to address it seamlessly through either our bootstrap script ( > bootstrap.sh in the root ) or within CMAKE. Obviously we don't want to > alienate Debian users. > > > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844018 > [2] https://issues.apache.org/jira/browse/MINIFICPP-400. > [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858398 > > On Tue, Feb 13, 2018 at 8:28 AM, Arne Oslebo <[email protected]> > wrote: > >> Marc, >> >> I'm running it on Debian Stretch. libssl version might be the problem. I >> see that both libssl1.0.2 and libssl1.1 are installed. I took another look >> at the build log and found this: >> /usr/bin/ld: warning: libssl.so.1.0.2, needed by >> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, >> may conflict with libssl.so.1.1 >> /usr/bin/ld: warning: libcrypto.so.1.0.2, needed by >> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, >> may conflict with libcrypto.so.1.1 >> >> I'll see if can remove one of the versions to avoid possible conflict. >> >> The full backtrace log is attached and thanks for looking into this, >> >> Arne >> >> >> >> On 13/02/2018 12:55, Marc wrote: >> >> Arne, >> Sorry for your troubles! Can you give me some insight into your distro? >> >> I've been unable to replicate the issue on OSX, u16, and arch...but >> all are running a different version of OpenSSL. What version of OpenSSL are >> you running and on what distro? I'll be happy to try it to track this down. >> >> Additionally, do you have the log file you can pass on? The reason I >> ask is that the line specified in gdb is a log statement with usage of >> constructs that are inherent to libstdc++, so they won't cause a memory >> error. The log file should help identify this and give me insight into what >> occurred just before the error. >> >> Thanks, >> Marc >> >> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo <[email protected]> >> wrote: >> >>> Hello Mark, >>> >>> thanks for the update. I tried running master from github but >>> unfortunately I now get a segmentation fault: >>> >>> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault. >>> 0x00007ffff777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 >>> (gdb) bt full >>> #0 0x00007ffff777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs >>> sl.so.1.1 >>> No symbol table info available. >>> #1 0x00007ffff7799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs >>> sl.so.1.1 >>> No symbol table info available. >>> #2 0x00007ffff777f2f6 in SSL_CTX_use_certificate () from >>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1 >>> No symbol table info available. >>> #3 0x00007ffff777f6c0 in SSL_CTX_use_certificate_file () from >>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1 >>> No symbol table info available. >>> #4 0x0000555555ef91cb in org::apache::nifi::minifi::con >>> trollers::SSLContextService::configure_ssl_context >>> (this=0x5555569948b0, ctx=0x555556a28430) at /usr/local/src/nifi-minifi-cpp >>> /extensions/http-curl/../../libminifi/include/controllers/SS >>> LContextService.h:165 >>> retp = 1 >>> #5 0x0000555555ef9be4 in org::apache::nifi::minifi::uti >>> ls::HTTPClient::configure_ssl_context (curl=0x555556a149e0, >>> ctx=0x555556a28430, param=0x5555569948b0) at /usr/local/src/nifi-minifi-cpp >>> /extensions/http-curl/client/HTTPClient.h:177 >>> ssl_context_service = 0x5555569948b0 >>> >>> Any idea what the problem might be? >>> >>> My full config.yml: >>> >>> Flow Controller: >>> name: MiNiFi Flow >>> Controller Services: >>> - name: SSLServiceName >>> id: 2438e3c8-015a-1000-79ca-83af40ec1974 >>> class: SSLContextService >>> Properties: >>> Client Certificate: /opt/minifi/conf/client.pem >>> Private Key: /opt/minifi/conf/client.key >>> Passphrase: secret >>> CA Certificate: /opt/minifi/conf/nifi-cert.pem >>> Processors: >>> - id: cecb1868-9e5a-3e6c-0000-000000000000 >>> name: TailFile >>> class: org.apache.nifi.processors.standard.TailFile >>> max concurrent tasks: 1 >>> scheduling strategy: TIMER_DRIVEN >>> scheduling period: 0 sec >>> penalization period: 30 sec >>> yield period: 1 sec >>> run duration nanos: 0 >>> auto-terminated relationships list: [] >>> Properties: >>> File Location: Local >>> File to Tail: /tmp/test.log >>> Initial Start Position: Beginning of File >>> Rolling Filename Pattern: >>> tail-base-directory: >>> tail-mode: Single file >>> tailfile-lookup-frequency: 10 minutes >>> tailfile-maximum-age: 24 hours >>> tailfile-recursive-lookup: 'false' >>> Connections: >>> - id: 76ad4bc4-6557-3e23-0000-000000000000 >>> name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043 >>> source id: cecb1868-9e5a-3e6c-0000-000000000000 >>> source relationship names: >>> - success >>> destination id: 56ae5abc-0161-1000-aa9e-c340d726e043 >>> max work queue size: 10000 >>> max work queue data size: 1 GB >>> flowfile expiration: 0 sec >>> queue prioritizer class: '' >>> Remote Processing Groups: >>> - id: 3a25e1a3-c1b2-3e78-0000-000000000000 >>> name: '' >>> url: https://w.x.y.z:8443/nifi >>> comment: '' >>> timeout: 30 sec >>> yield period: 10 sec >>> transport protocol: RAW >>> proxy host: '' >>> proxy port: '' >>> proxy user: '' >>> proxy password: '' >>> local network interface: '' >>> Input Ports: >>> - id: 56ae5abc-0161-1000-aa9e-c340d726e043 >>> name: Minifi >>> comment: '' >>> max concurrent tasks: 1 >>> use compression: false >>> Properties: >>> Port: 10443 >>> SSL Context Service: SSLServiceName >>> Host Name: w.x.y.z >>> Output Ports: [] >>> Provenance Reporting: >>> >>> >>> >>> On 09/02/2018 20:18, Marc wrote: >>> >>> Arne, >>> I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263 >>> to address these issues. >>> >>> On Fri, Feb 9, 2018 at 8:38 AM, Marc <[email protected]> wrote: >>> >>>> Arne, >>>> Evidently the HTTPClient relies on an SSL Context Service. Try the >>>> following configuration in the config.yml file, where you define the >>>> context service and reference it from the RPG. Let me know if that works >>>> for you! >>>> >>>> Additionally, I think you pointed out an inconsistency where we can >>>> improve the configuration and documentation. I've created >>>> https://issues.apache.org/jira/browse/MINIFICPP-396 and will take care >>>> of that ASAP. Thanks >>>> very much for identifying this! >>>> >>>> Remote Processing Groups: >>>> - name: NiFi Flow >>>> id: 2438e3c8-015a-1000-79ca-83af40ec1998 >>>> url: https://127.0.0.1:8383/nifi >>>> timeout: 30 secs >>>> yield period: 5 sec >>>> Input Ports: >>>> - id: 2438e3c8-015a-1000-79ca-83af40ec1999 >>>> name: fromnifi >>>> max concurrent tasks: 1 >>>> Properties: >>>> Port: 10443 >>>> SSL Context Service: SSLMe >>>> Host Name: 127.0.0.1 >>>> Output Ports: >>>> - id: ac82e521-015c-1000-2b21-41279516e19a >>>> name: tominifi >>>> max concurrent tasks: 2 >>>> Properties: >>>> Port: 10443 >>>> SSL Context Service: SSLMe >>>> Host Name: 127.0.0.1 >>>> >>>> Controller Services: >>>> - name: SSLMe >>>> id: 2438e3c8-015a-1000-79ca-83af40ec1974 >>>> class: SSLContextService >>>> Properties: >>>> Client Certificate: /opt/minifi/conf/client.pem >>>> Private Key: /opt/minifi/conf/client.key >>>> Passphrase: /opt/minifi/conf/password >>>> CA Certificate certificate: /opt/minifi/conf/nifi-cert.pem >>>> >>>> On Fri, Feb 9, 2018 at 5:54 AM, Arne Oslebo <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm trying to set up secure communication between minifi-cpp 0.4.0 and >>>>> nifi, but unfortunately it fails with the following errors: >>>>> >>>>> [org::apache::nifi::minifi::utils::HTTPClient] [error] >>>>> curl_easy_perform() failed SSL connect error >>>>> [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error] >>>>> ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed >>>>> >>>>> I looked quickly at the code and it seems the problem is that >>>>> HTTPClient >>>>> never calls configure_secure_connection and therefor never presents a >>>>> client certificate to nifi. >>>>> >>>>> The config.yml file defines a TailFail that send data directly to a >>>>> remote process group. >>>>> >>>>> My minifi.properties file: >>>>> nifi.version=0.1.0 >>>>> nifi.flow.configuration.file=/opt/minifi/conf/config.yml >>>>> nifi.administrative.yield.duration=30 sec >>>>> nifi.bored.yield.duration=10 millis >>>>> nifi.provenance.repository.directory.default=/opt/minifi/pro >>>>> venance_repository >>>>> nifi.provenance.repository.max.storage.time=1 MIN >>>>> nifi.provenance.repository.max.storage.size=1 MB >>>>> nifi.remote.input.secure=true >>>>> nifi.https.need.ClientAuth=true >>>>> nifi.https.client.certificate=/opt/minifi/conf/client.pem >>>>> nifi.https.client.private.key=/opt/minifi/conf/client.key >>>>> nifi.https.client.pass.phrase=/opt/minifi/conf/password >>>>> nifi.https.client.ca.certificate=/opt/minifi/conf/nifi-cert.pem >>>>> controller.socket.host=localhost >>>>> controller.socket.port=9998 >>>>> >>>>> Certificates and key are correct and have been verified using curl from >>>>> the command line. Are there any other things I need to do to get minifi >>>>> to set up a secure connection? As far as I understand the "Security >>>>> Properties:" in config.yml is only used by the java version of minifi? >>>>> >>>>> Thanks, >>>>> Arne >>>>> >>>>> >>>> >>> >>> >> >> >
