Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh

['sanjay...@google.com' via grpc.io]

I looked at the logs and I can confirm that the client is not using mTLS 
because Istio didn't provide the right configuration. Let me explain:

>From your server-mtls-strict.log I see this 

  "transportSocket": {
"name": "envoy.transport_sockets.tls",
"typedConfig": {
  "@type": 
"type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
  "commonTlsContext": {
"combinedValidationContext": {
  "defaultValidationContext": {
  },
  "validationContextCertificateProviderInstance": {
"instanceName": "default",
"certificateName": "ROOTCA"
  }
},
"tlsCertificateCertificateProviderInstance": {
  "instanceName": "default",
  "certificateName": "default"
}
  },
  "requireClientCertificate": true
}
  },
  "name": "inbound-mtls"

In the Listener resource received (see timestamp 16:47:54.070)

However I don't see a similar security configuration on the client side - 
which is a bit complex I admin. In the client-mtls-strict.log I see a 
Cluster resource response:

{
  "versionInfo": "2023-05-26T16:47:41Z/749",
  "resources": [{
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
  "edsConfig": {
"ads": {
}
  },
  "serviceName": 
"outbound|8443||server.kotlin-grpc-xds.svc.cluster.local"
}
  }],
  "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255",
  "controlPlane": {
"identifier": 
"{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}"
  }
}

which has no security/mTLS configuration. I would expect a config similar 
to the transportSocket snippet I described above.

So the error is on the Istio side: whether a bug in Istio or a user-error I 
can't tell. I hope you are able to figure it out?



On Friday, May 26, 2023 at 10:53:34 AM UTC-7 Wesley Hartford wrote:

> I've attached a copy of the log files with xds logging set to trace for an 
> execution of the client and server with istio's mtls mode set to STRICT and 
> PERMISSIVE. My interpretation of these logs is:
>
> In PERMISSIVE mode, neither client nor server is trying to use any type of 
> TLS; they're both using plain text and can interact just fine. I was under 
> the impression that PERMISSIVE mode would use mTLS, but reading the istio 
> docs, it's a little ambiguous, so I guess this is a correct behaviour.
>
> In STRICT mode, it looks like the server is using the supplied TLS key 
> material to accept mTLS connections, but it doesn't look like the client is 
> making any attempt to use TLS.
>
> I'm working on your other suggestions, but thought I should update you 
> with the logs before spending too much time on them. 
>
> On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare  
> wrote:
>
>> (adding grpc.io group back)
>>
>> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford  
>> wrote:
>>
>>> Hi,
>>>
>>> My suggestion that the connection was falling back to insecure was not 
>>> evidence based, I'm still trying to wrap my head around how all this is 
>>> working.
>>>
>>
>> Okay.
>>  
>>
>>>
>>> The target address on the client side is using the xds:/// prefix.
>>>
>>> I've enabled trace level logging on the io.grpc.xds logger but I'm not 
>>> seeing any additional log messages, have I missed something? I'm using 
>>> slf4j and logback and have the SLF4JBridgeHandler installed.
>>>
>>
>> The code uses Java util logging so you can just something like 
>> -Djava.util.logging.config.file=/logging.properties in your Java 
>> invocation command line and enable the logger for io.grpc.xds  .
>>  
>>
>>>
>>> The grpc-bootstrap.json file seems reasonable, though I don't know just 
>>> what it all means (I've attached the content). The three pem files 
>>> referenced in certificate_providers point to real files containing 
>>> apparently valid PEM content.
>>>
>>> You've mentioned a couple times enabling vs. disabling mTLS, are you 
>>> referring to some specific setting on the client and/or server, or in istio 
>>> somewhere? My understanding has been that with the Xds server and channel, 
>>> both will use mTLS unless I specifically set the mtls mode to DISABLE in a 
>>> PeerAuthentication resource, which I haven't done. I've experimented with 
>>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple 
>>> as not enabling mTLS somewhere?
>>>
>>
>> In your first post you said "I got everything working without too much 
>> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got 
>> the impression 

Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh

[Wesley Hartford]

I've attached a copy of the log files with xds logging set to trace for an
execution of the client and server with istio's mtls mode set to STRICT and
PERMISSIVE. My interpretation of these logs is:

In PERMISSIVE mode, neither client nor server is trying to use any type of
TLS; they're both using plain text and can interact just fine. I was under
the impression that PERMISSIVE mode would use mTLS, but reading the istio
docs, it's a little ambiguous, so I guess this is a correct behaviour.

In STRICT mode, it looks like the server is using the supplied TLS key
material to accept mTLS connections, but it doesn't look like the client is
making any attempt to use TLS.

I'm working on your other suggestions, but thought I should update you with
the logs before spending too much time on them.

On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare 
wrote:

> (adding grpc.io group back)
>
> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford 
> wrote:
>
>> Hi,
>>
>> My suggestion that the connection was falling back to insecure was not
>> evidence based, I'm still trying to wrap my head around how all this is
>> working.
>>
>
> Okay.
>
>
>>
>> The target address on the client side is using the xds:/// prefix.
>>
>> I've enabled trace level logging on the io.grpc.xds logger but I'm not
>> seeing any additional log messages, have I missed something? I'm using
>> slf4j and logback and have the SLF4JBridgeHandler installed.
>>
>
> The code uses Java util logging so you can just something like
> -Djava.util.logging.config.file=/logging.properties in your Java
> invocation command line and enable the logger for io.grpc.xds  .
>
>
>>
>> The grpc-bootstrap.json file seems reasonable, though I don't know just
>> what it all means (I've attached the content). The three pem files
>> referenced in certificate_providers point to real files containing
>> apparently valid PEM content.
>>
>> You've mentioned a couple times enabling vs. disabling mTLS, are you
>> referring to some specific setting on the client and/or server, or in istio
>> somewhere? My understanding has been that with the Xds server and channel,
>> both will use mTLS unless I specifically set the mtls mode to DISABLE in a
>> PeerAuthentication resource, which I haven't done. I've experimented with
>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple
>> as not enabling mTLS somewhere?
>>
>
> In your first post you said "I got everything working without too much
> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got
> the impression that you got everything working in plaintext (without
> enabling mTLS) and then you enabled mTLS via Istio which is when you saw
> connection problems. I was just referring to the same - you enable mTLS in
> Istio security policy.
>
> I suggest couple of additional tests to troubleshoot this:
>
> - instead of using Xds Channel and Server credentials use the Tls Channel
> and Server credentials with the same files provided by Istio (under
> /var/lib/istio/data). You can check out the doc at
> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html
> and
> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html
> . Note that even if you are using XDS for load balancing, service discovery
> you can use Tls credentials for security.
>
> - if you can try a Golang (or C++) example with your Istio setup and
> verify mTLS is working with those examples.
>
> Hope that helps.
>
>
>
>>
>> Thanks again,
>>
>> Wesley
>>
>> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io <
>> grpc-io@googlegroups.com> wrote:
>>
>>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:
>>>
>>> Thanks for getting back to me, Sanjay. As far as I can tell, my client
>>> and server are both using the appropriate Xds credentials:
>>> The client code is at
>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
>>>   Grpc.newChannelBuilder(targetUrl,
>>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()
>>>
>>> The server code is at
>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
>>>   XdsServerBuilder.forPort(8443,
>>> XdsServerCredentials.create(InsecureServerCredentials.create()))
>>>
>>> The insecure credentials provided to both a fallback, and it looks like
>>> the sample you linked is doing the same thing.
>>>
>>>
>>> Yes, you are right - Xds credentials are being correctly used so that's
>>> not an issue.
>>>
>>> I'm not sure why, but I'm guessing that the secure connection is failing
>>> and it is falling back to insecure.
>>>
>>>
>>> No, that cannot be the case. As described here
>>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
>>> fallback credentials are *not* 

Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh

['Sanjay Pujare' via grpc.io]

(adding grpc.io group back)

On Wed, May 24, 2023 at 2:57 PM Wesley Hartford 
wrote:

> Hi,
>
> My suggestion that the connection was falling back to insecure was not
> evidence based, I'm still trying to wrap my head around how all this is
> working.
>

Okay.


>
> The target address on the client side is using the xds:/// prefix.
>
> I've enabled trace level logging on the io.grpc.xds logger but I'm not
> seeing any additional log messages, have I missed something? I'm using
> slf4j and logback and have the SLF4JBridgeHandler installed.
>

The code uses Java util logging so you can just something like
-Djava.util.logging.config.file=/logging.properties in your Java invocation
command line and enable the logger for io.grpc.xds  .


>
> The grpc-bootstrap.json file seems reasonable, though I don't know just
> what it all means (I've attached the content). The three pem files
> referenced in certificate_providers point to real files containing
> apparently valid PEM content.
>
> You've mentioned a couple times enabling vs. disabling mTLS, are you
> referring to some specific setting on the client and/or server, or in istio
> somewhere? My understanding has been that with the Xds server and channel,
> both will use mTLS unless I specifically set the mtls mode to DISABLE in a
> PeerAuthentication resource, which I haven't done. I've experimented with
> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple
> as not enabling mTLS somewhere?
>

In your first post you said "I got everything working without too much
trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got the
impression that you got everything working in plaintext (without enabling
mTLS) and then you enabled mTLS via Istio which is when you saw connection
problems. I was just referring to the same - you enable mTLS in Istio
security policy.

I suggest couple of additional tests to troubleshoot this:

- instead of using Xds Channel and Server credentials use the Tls Channel
and Server credentials with the same files provided by Istio (under
/var/lib/istio/data). You can check out the doc at
https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html
and
https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html
. Note that even if you are using XDS for load balancing, service discovery
you can use Tls credentials for security.

- if you can try a Golang (or C++) example with your Istio setup and verify
mTLS is working with those examples.

Hope that helps.



>
> Thanks again,
>
> Wesley
>
> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io <
> grpc-io@googlegroups.com> wrote:
>
>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:
>>
>> Thanks for getting back to me, Sanjay. As far as I can tell, my client
>> and server are both using the appropriate Xds credentials:
>> The client code is at
>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
>>   Grpc.newChannelBuilder(targetUrl,
>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()
>>
>> The server code is at
>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
>>   XdsServerBuilder.forPort(8443,
>> XdsServerCredentials.create(InsecureServerCredentials.create()))
>>
>> The insecure credentials provided to both a fallback, and it looks like
>> the sample you linked is doing the same thing.
>>
>>
>> Yes, you are right - Xds credentials are being correctly used so that's
>> not an issue.
>>
>> I'm not sure why, but I'm guessing that the secure connection is failing
>> and it is falling back to insecure.
>>
>>
>> No, that cannot be the case. As described here
>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
>> fallback credentials are *not* meant to be used when secure connection
>> fails. If you suspect that's happening can you provide some evidence (logs
>> etc)?
>>
>>
>> Based on the example you linked, the only other requirement is that the
>> GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by
>> the istio sidecar; kubectl describe pod shows that both the client and
>> server containers have two environment variables injected:
>>   GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT:  true
>>   GRPC_XDS_BOOTSTRAP:
>>  /etc/istio/proxy/grpc-bootstrap.json
>>
>>
>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) is
>> required not just for security but overall XDS support. Based on your
>> observations in the first email it looks like client and server are able to
>> communicate based on xds configuration so that part must be working. Just
>> to confirm your client did use the "xds:///" prefix to access your service,
>> right?
>>
>> If you are certain that XDS is working 

Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh

['sanjay...@google.com' via grpc.io]

On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:

Thanks for getting back to me, Sanjay. As far as I can tell, my client and 
server are both using the appropriate Xds credentials:
The client code is at 
https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
  Grpc.newChannelBuilder(targetUrl, 
XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()

The server code is at 
https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
  XdsServerBuilder.forPort(8443, 
XdsServerCredentials.create(InsecureServerCredentials.create()))

The insecure credentials provided to both a fallback, and it looks like the 
sample you linked is doing the same thing.


Yes, you are right - Xds credentials are being correctly used so that's not 
an issue.

I'm not sure why, but I'm guessing that the secure connection is failing 
and it is falling back to insecure.


No, that cannot be the case. As described 
here 
https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
 
fallback credentials are *not* meant to be used when secure connection 
fails. If you suspect that's happening can you provide some evidence (logs 
etc)?
 

Based on the example you linked, the only other requirement is that the 
GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by the 
istio sidecar; kubectl describe pod shows that both the client and server 
containers have two environment variables injected:
  GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT:  true
  GRPC_XDS_BOOTSTRAP: 
 /etc/istio/proxy/grpc-bootstrap.json


The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) is 
required not just for security but overall XDS support. Based on your 
observations in the first email it looks like client and server are able to 
communicate based on xds configuration so that part must be working. Just 
to confirm your client did use the "xds:///" prefix to access your service, 
right? 

If you are certain that XDS is working for load balancing and plaintext 
communication but breaks only when mTLS is enabled it might be useful to 
enable debug logging on both client and server side (for the io.grpc.xds 
package and below). 

Also you can check the contents of /etc/istio/proxy/grpc-bootstrap.json esp. 
the certificate_providers part and confirm that the 3 files provided in the 
config have valid certificate material. Let me know if you need help 
debugging that part.

 

There are only two warning lines being logged from both the client and the 
server:

14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown 
channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown 
channel option 
'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT'
 
for channel '[id: 0xba433026]'


I doubt this is an issue. Just to confirm you see these messages also when 
mTLS is not enabled, right?

 

Do you know of anything else I might be missing that is required for a 
secure connection?

Thanks,

Wesley

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com.

Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh

[Wesley Hartford]

Thanks for getting back to me, Sanjay. As far as I can tell, my client and
server are both using the appropriate Xds credentials:
The client code is at
https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
  Grpc.newChannelBuilder(targetUrl,
XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()

The server code is at
https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
  XdsServerBuilder.forPort(8443,
XdsServerCredentials.create(InsecureServerCredentials.create()))

The insecure credentials provided to both a fallback, and it looks like the
sample you linked is doing the same thing. I'm not sure why, but I'm
guessing that the secure connection is failing and it is falling back to
insecure. Based on the example you linked, the only other requirement is
that the GRPC_XDS_BOOTSTRAP environment variable is set, which is being
done by the istio sidecar; kubectl describe pod shows that both the client
and server containers have two environment variables injected:
  GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT:  true
  GRPC_XDS_BOOTSTRAP:
 /etc/istio/proxy/grpc-bootstrap.json

There are only two warning lines being logged from both the client and the
server:

14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown
channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
14:51:55.314 [main] WARN  i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown
channel option
'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT'
for channel '[id: 0xba433026]'

Do you know of anything else I might be missing that is required for a
secure connection?

Thanks,

Wesley


On Tue, May 23, 2023 at 11:10 PM 'sanjay...@google.com' via grpc.io <
grpc-io@googlegroups.com> wrote:

> On Wednesday, May 17, 2023 at 11:07:43 AM UTC-7 Wesley Hartford wrote:
>
> ...
> What doesn't seem right:
>
>- A server interceptor reports that ServerCall.getSecurityLevel()
>returns NONE,
>
>
> Seems right when you are using InsecureChannelCredentials i.e. plaintext.
>
>
>
>- When I configure Istio to enforce STRICT mTLS via a namespace wide
>PeerAuthentication resource, the client's connection to the server fails
>with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for
>priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1]
>
>
> You will have to modify the client code to use XdsCredentials as described
> in
> https://github.com/grpc/grpc-java/tree/master/examples/example-xds#run-the-example-with-xds-credentials
> . I am assuming the server is using XdsServerCredentials.
>
>
> Is this the expected behavior, or have I missed something?
>
> Thanks for any insight you might have.
>
> Wesley
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "grpc.io" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/3e513ace-30e7-4a7d-8023-dde3a904be3cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CA%2B%2B-c5w3J6ROX1NRmHFmhficeh-YRDe0Y1b6EUR%2B40hYDhEpjw%40mail.gmail.com.

[grpc-io] Re: Proxyless gRPC services in Istio mesh

['sanjay...@google.com' via grpc.io]

On Wednesday, May 17, 2023 at 11:07:43 AM UTC-7 Wesley Hartford wrote:

...
What doesn't seem right:

   - A server interceptor reports that ServerCall.getSecurityLevel() 
   returns NONE,
   

Seems right when you are using InsecureChannelCredentials i.e. plaintext.
 


   - When I configure Istio to enforce STRICT mTLS via a namespace wide 
   PeerAuthentication resource, the client's connection to the server fails 
   with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for 
   priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1]


You will have to modify the client code to use XdsCredentials as described 
in 
https://github.com/grpc/grpc-java/tree/master/examples/example-xds#run-the-example-with-xds-credentials
 
. I am assuming the server is using XdsServerCredentials. 


Is this the expected behavior, or have I missed something?

Thanks for any insight you might have.

Wesley

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/3e513ace-30e7-4a7d-8023-dde3a904be3cn%40googlegroups.com.