I looked at the logs and I can confirm that the client is not using mTLS
because Istio didn't provide the right configuration. Let me explain:
>From your server-mtls-strict.log I see this
"transportSocket": {
"name": "envoy.transport_sockets.tls",
"typedConfig": {
"@type":
"type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
"commonTlsContext": {
"combinedValidationContext": {
"defaultValidationContext": {
},
"validationContextCertificateProviderInstance": {
"instanceName": "default",
"certificateName": "ROOTCA"
}
},
"tlsCertificateCertificateProviderInstance": {
"instanceName": "default",
"certificateName": "default"
}
},
"requireClientCertificate": true
}
},
"name": "inbound-mtls"
In the Listener resource received (see timestamp 16:47:54.070)
However I don't see a similar security configuration on the client side -
which is a bit complex I admin. In the client-mtls-strict.log I see a
Cluster resource response:
{
"versionInfo": "2023-05-26T16:47:41Z/749",
"resources": [{
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {
}
},
"serviceName":
"outbound|8443||server.kotlin-grpc-xds.svc.cluster.local"
}
}],
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255",
"controlPlane": {
"identifier":
"{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}"
}
}
which has no security/mTLS configuration. I would expect a config similar
to the transportSocket snippet I described above.
So the error is on the Istio side: whether a bug in Istio or a user-error I
can't tell. I hope you are able to figure it out?
On Friday, May 26, 2023 at 10:53:34 AM UTC-7 Wesley Hartford wrote:
> I've attached a copy of the log files with xds logging set to trace for an
> execution of the client and server with istio's mtls mode set to STRICT and
> PERMISSIVE. My interpretation of these logs is:
>
> In PERMISSIVE mode, neither client nor server is trying to use any type of
> TLS; they're both using plain text and can interact just fine. I was under
> the impression that PERMISSIVE mode would use mTLS, but reading the istio
> docs, it's a little ambiguous, so I guess this is a correct behaviour.
>
> In STRICT mode, it looks like the server is using the supplied TLS key
> material to accept mTLS connections, but it doesn't look like the client is
> making any attempt to use TLS.
>
> I'm working on your other suggestions, but thought I should update you
> with the logs before spending too much time on them.
>
> On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare <sanjay...@google.com>
> wrote:
>
>> (adding grpc.io group back)
>>
>> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca>
>> wrote:
>>
>>> Hi,
>>>
>>> My suggestion that the connection was falling back to insecure was not
>>> evidence based, I'm still trying to wrap my head around how all this is
>>> working.
>>>
>>
>> Okay.
>>
>>
>>>
>>> The target address on the client side is using the xds:/// prefix.
>>>
>>> I've enabled trace level logging on the io.grpc.xds logger but I'm not
>>> seeing any additional log messages, have I missed something? I'm using
>>> slf4j and logback and have the SLF4JBridgeHandler installed.
>>>
>>
>> The code uses Java util logging so you can just something like
>> -Djava.util.logging.config.file=/logging.properties in your Java
>> invocation command line and enable the logger for io.grpc.xds .
>>
>>
>>>
>>> The grpc-bootstrap.json file seems reasonable, though I don't know just
>>> what it all means (I've attached the content). The three pem files
>>> referenced in certificate_providers point to real files containing
>>> apparently valid PEM content.
>>>
>>> You've mentioned a couple times enabling vs. disabling mTLS, are you
>>> referring to some specific setting on the client and/or server, or in istio
>>> somewhere? My understanding has been that with the Xds server and channel,
>>> both will use mTLS unless I specifically set the mtls mode to DISABLE in a
>>> PeerAuthentication resource, which I haven't done. I've experimented with
>>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple
>>> as not enabling mTLS somewhere?
>>>
>>
>> In your first post you said "I got everything working without too much
>> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got
>> the impression that you got everything working in plaintext (without
>> enabling mTLS) and then you enabled mTLS via Istio which is when you saw
>> connection problems. I was just referring to the same - you enable mTLS in
>> Istio security policy.
>>
>> I suggest couple of additional tests to troubleshoot this:
>>
>> - instead of using Xds Channel and Server credentials use the Tls Channel
>> and Server credentials with the same files provided by Istio (under
>> /var/lib/istio/data). You can check out the doc at
>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html
>> and
>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html
>> . Note that even if you are using XDS for load balancing, service discovery
>> you can use Tls credentials for security.
>>
>> - if you can try a Golang (or C++) example with your Istio setup and
>> verify mTLS is working with those examples.
>>
>> Hope that helps.
>>
>>
>>
>>>
>>> Thanks again,
>>>
>>> Wesley
>>>
>>> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io <
>>> grp...@googlegroups.com> wrote:
>>>
>>>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:
>>>>
>>>> Thanks for getting back to me, Sanjay. As far as I can tell, my client
>>>> and server are both using the appropriate Xds credentials:
>>>> The client code is at
>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
>>>> Grpc.newChannelBuilder(targetUrl,
>>>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()
>>>>
>>>> The server code is at
>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
>>>> XdsServerBuilder.forPort(8443,
>>>> XdsServerCredentials.create(InsecureServerCredentials.create()))
>>>>
>>>> The insecure credentials provided to both a fallback, and it looks like
>>>> the sample you linked is doing the same thing.
>>>>
>>>>
>>>> Yes, you are right - Xds credentials are being correctly used so that's
>>>> not an issue.
>>>>
>>>> I'm not sure why, but I'm guessing that the secure connection is
>>>> failing and it is falling back to insecure.
>>>>
>>>>
>>>> No, that cannot be the case. As described here
>>>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
>>>>
>>>> fallback credentials are *not* meant to be used when secure connection
>>>> fails. If you suspect that's happening can you provide some evidence (logs
>>>> etc)?
>>>>
>>>>
>>>> Based on the example you linked, the only other requirement is that the
>>>> GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by
>>>> the istio sidecar; kubectl describe pod shows that both the client and
>>>> server containers have two environment variables injected:
>>>> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true
>>>> GRPC_XDS_BOOTSTRAP:
>>>> /etc/istio/proxy/grpc-bootstrap.json
>>>>
>>>>
>>>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable)
>>>> is required not just for security but overall XDS support. Based on your
>>>> observations in the first email it looks like client and server are able
>>>> to
>>>> communicate based on xds configuration so that part must be working. Just
>>>> to confirm your client did use the "xds:///" prefix to access your
>>>> service,
>>>> right?
>>>>
>>>> If you are certain that XDS is working for load balancing and plaintext
>>>> communication but breaks only when mTLS is enabled it might be useful to
>>>> enable debug logging on both client and server side (for the io.grpc.xds
>>>> package and below).
>>>>
>>>> Also you can check the contents of
>>>> /etc/istio/proxy/grpc-bootstrap.json esp. the certificate_providers
>>>> part and confirm that the 3 files provided in the config have valid
>>>> certificate material. Let me know if you need help debugging that part.
>>>>
>>>>
>>>>
>>>> There are only two warning lines being logged from both the client and
>>>> the server:
>>>>
>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>> Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>> Unknown channel option
>>>> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT'
>>>>
>>>> for channel '[id: 0xba433026]'
>>>>
>>>>
>>>> I doubt this is an issue. Just to confirm you see these messages also
>>>> when mTLS is not enabled, right?
>>>>
>>>>
>>>>
>>>> Do you know of anything else I might be missing that is required for a
>>>> secure connection?
>>>>
>>>> Thanks,
>>>>
>>>> Wesley
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "grpc.io" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> grpc-io+u...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/8e2dc818-f08c-4c77-ac3e-65c68c3fe6b6n%40googlegroups.com.
