martroben opened a new issue, #5696: URL: https://github.com/apache/arrow-rs/issues/5696
# Which part is this question about object_store features # Describe your question Is there some setting for object_store that would enable accessing Azure storage if the storage is using a self-signed certificate? # Additional context I would like to use the Python polars library on the [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric) platform. [Polars](https://docs.pola.rs/py-polars/html/reference/) uses [delta-rs](https://delta-io.github.io/delta-rs/) which in turn uses object_store to interact with OneLake. ([OneLake](https://learn.microsoft.com/en-us/fabric/onelake/onelake-api-parity) is the Azure Data Lake Storage Gen2 flavor used for storage in Fabric). Everything works fine when accessing OneLake from a local device. However, when I try to use Polars in a Fabric notebook, I get the following error: `error trying to connect: invalid peer certificate: Other(CaUsedAsEndEntity)`. I checked the certificate with `openssl s_client -connect onelake.blob.fabric.microsoft.com:443 -showcerts | openssl x509 -text` in the Fabric Notebook environment. It turns out that onelake.blob.fabric.microsoft.com indeed uses a self-signed Certificate Authority (CA) certificate for connections coming from Fabric Notebooks. If I run the same check from a local device, the server provides a normal End Entity certificate. I suspect that the error is caused by some upstream certificate validator function in [webpki](https://docs.rs/webpki/latest/webpki/), which is in turn used by [rustls](https://docs.rs/rustls/latest/rustls/). I don't speak rust very well, but I assume that object_store also uses rustls. The issue [has](https://github.com/briansmith/webpki/issues/114) been [raised](https://github.com/rustls/rustls/issues/143) before [in](https://github.com/rustls/rustls/issues/772) webpki and rustls repos, but the maintainers of both have made it clear that they're [not](https://github.com/briansmith/webpki/pull/170) going [to](https://github.com/rustls/rustls/blob/513e374b2e2ce9f1fb57ac78ab3ca053afc8f133/rustls/src/manual/features.rs#L46) provide an out-of-the-box option for accepting connections that are trying to use CA certificate as EE. rustls includes an option to [implement](https://github.com/rustls/rustls/pull/1368) [custom](https://github.com/rustls/rustls/issues/1360) [certificate](https://github.com/rustls/rustls/blob/992e2364a006b2e84a8cf6a7c3eaf0bdb773c9de/examples/src/bin/tlsclient-mio.rs#L318) verifier. However, since I don't speak the language, I'm not sure if it is already included in object_store - and if not, could it be done? The only check that would need to be omitted is the one asserting that `CA` needs to be `FALSE` in the certificate: ``` Certificate: Data: X509v3 extensions: X509v3 Basic Constraints: critical CA:FALSE ``` I have also opened a support case with the Microsoft Fabric team, asking them to start using valid certificates (request #2404260050002071). However, if anybody's going to solve it, my money is on the open source community. As of now, rust-based libraries are basically unusable in Microsoft Fabric, which is unfortunate. PS: I'm aware of the [`allow_invalid_certificates`](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_allow_invalid_certificates) option and it does fix the problem. But my love for polars isn't big enough to start using unsecure connections in production. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
