Is this an issue with the flink server or with how gRPC is configured on the client? The code in question "grpc.local_channel_credentials" makes me think this has to do with how the client is configured (maybe a client certificate).
Are there differences in the client in how the 3 gRPC clients are constructed? i.e. is it possible job service and artifact service are always using insecure? There appear to be some differences in the flink job server with respect to how the gRPC services are constructed so maybe that explains why secure channels don't work for the expansion service? https://github.com/apache/beam/blob/5fb31eb11c5ad967ca3e674529f943527abb0872/runners/java-job-service/src/main/java/org/apache/beam/runners/jobsubmission/JobServerDriver.java#L265 I noticed for example that the gRPC services for the artifact and job service bind all devices by default but the expansion service just binds localhost. J On Fri, Aug 27, 2021 at 4:24 PM Chamikara Jayalath <[email protected]> wrote: > Yeah, but I'd rather try to make the current path work with secure > channels first. Is the Flink job server limited to insecure > channels somehow ? > > On Fri, Aug 27, 2021 at 4:01 PM Kyle Weaver <[email protected]> wrote: > >> Can we make the choice of secure/insecure channels configurable? >> >> On Fri, Aug 27, 2021 at 3:57 PM Chamikara Jayalath <[email protected]> >> wrote: >> >>> I don't know why this is failing but the move to secure channels was >>> done due to a real failure in a different environment. >>> >>> On Fri, Aug 27, 2021 at 12:44 PM Jeremy Lewi <[email protected]> >>> wrote: >>> >>>> Hi Folks, >>>> >>>> I'm running Beam 2.33 using a JobServer on Flink on Kubernetes. >>>> >>>> My client is unable to connect to the expansion service running on the >>>> job server. Connecting to the other services works fine and I can submit >>>> jobs that don't require the expansion service. >>>> >>>> When I debugged this it looks like the problem is that the code ends up >>>> trying to create a secure gRPC channel for the expansion service. I traced >>>> the problem to this line >>>> >>>> >>>> https://github.com/apache/beam/blob/4bd217c0d29f88b0b1e0b8211f7bd13a63a6157a/sdks/python/apache_beam/transforms/external.py#L328 >>>> >>>> When I hacked the code to use an insecure channel it worked. Is this >>>> working as intended? >>>> >>>> Thanks >>>> J >>>> >>>
