Thanks David for the information. My main issue is that we are doing massive web scraping (over 400k websites and growing) and I can not just add each certificate manually. I can probably automate most of it but I wanted to see what options were available to me.
Thanks again. I will look into this. Jean-Sébastien Vachon Co-Founder & Architect Brizo Data, Inc. www.brizodata.com<https://outlook.office365.com/mail/options/mail/messageContent/www.brizodata.com> ________________________________ From: David Handermann <[email protected]> Sent: Friday, March 4, 2022 9:16 AM To: [email protected] <[email protected]> Subject: Re: InvokeHTTP vs invalid SSL certificates Thanks for raising this question. The InvokeHTTP processor relies on the OkHttp client library, which implements standard TLS handshaking and hostname verification as described in their documentation: https://square.github.io/okhttp/features/https/ There are many things that could make a certificate invalid for a specific connection. If the remote certificate is self-signed, it is possible to configure a NiFi SSL Context Service with a trust store that includes the self-signed certificate. If the remote certificate is expired, the remote server must be updated with a new certificate. If the remote certificate does not include a DNS Subject Alternative Name (SAN) matching the domain name that InvokeHTTP uses for the connection, the best solution is for the remote server to be updated with a new certificate containing a matching SAN. It is possible to configure OkHttp with a custom hostname verifier or trust manager that ignores some of these attributes, but this would require custom code that overrides the default behavior of InvokeHTTP. There have been some requests in the past for NiFi to implement support for a custom hostname verifier, but this approach breaks one of the fundamental aspects of TLS communication security. With that background, the potential solution depends on why InvokeHTTP considers the certificate invalid. Regards, David Handermann On Fri, Mar 4, 2022 at 6:59 AM Jean-Sebastien Vachon <[email protected]<mailto:[email protected]>> wrote: Hi all, what is the best way to deal with invalid SSL certificates when trying to open an URL using InvokeHTTP? Thanks Jean-Sébastien Vachon Co-Founder & Architect Brizo Data, Inc. www.brizodata.com<https://outlook.office365.com/mail/options/mail/messageContent/www.brizodata.com>
