iantowey commented on PR #25872: URL: https://github.com/apache/pulsar/pull/25872#issuecomment-4626270377
@lhotari Happy to share our context. We are still in the testing phase for this architecture, so some specifics might evolve, but this is the general direction we are heading. **Higher Level Goal** We run the Pulsar core components (Proxy, Broker, BookKeeper, Zookeeper, and the Functions Worker) on bare metal as Podman containers. Our Pulsar deployment relies heavily on connectors—approximately 1,000 individual connectors, some with a high degree of parallelism. Currently, the Function Worker hosts the functions via the process runtime setup. We want to move away from this setup for a number of reasons (e.g., changing Function Worker configurations requires us to cycle through all the FWs, bringing down all the functions on that FW). Our goal is to deploy the connectors on Kubernetes while continuing to maintain the FW on bare metal. By default, the worker attempts to reach function instances using internal Kubernetes DNS (e.g., `[function-name].[namespace].svc.cluster.local`). Because our worker runs outside the cluster, it cannot resolve these internal addresses. This PR adds flexibility to customize the domain suffix, routing the worker's gRPC traffic through an external Kubernetes Gateway that we have configured to map to the internal function services. **Automating Gateway Routing with Kyverno** We deploy a Kyverno `ClusterPolicy` that watches for the creation of these function K8s Services (e.g., matching the `pf-*` naming convention). As soon as a Service is created, Kyverno automatically generates a corresponding `GRPCRoute` resource. This auto-generated route binds the function's specific hostname (utilizing the custom domain suffix introduced in this PR) to the Gateway, seamlessly establishing the path from the external worker to the internal pod. **Kubernetes Authentication Configuration** You are exactly right regarding the `KUBECONFIG` workaround. We generate a kubeconfig file with the appropriate cluster details and client certificates for a `pulsar` namespace, mount that file into the Function Worker container, and set the `KUBECONFIG` environment variable to point to it. This allows the worker to authenticate successfully without needing explicit auth fields added to `functions_worker.yml`. **Security Context** Regarding the plaintext gRPC connections and metrics, we acknowledge this design and are deploying in alignment with the assumption of a trusted perimeter. The traffic between the external Function Worker and the Kubernetes Gateway occurs strictly within a secured, private network boundary (VPC/LAN), isolating it from untrusted traffic. To enforce this trusted perimeter at the ingress point, our design utilizes IP restrictions on the Gateway to explicitly drop any gRPC or metrics traffic that does not originate from the known, static IP addresses of our external Function Worker nodes. Thanks again for the review and merge! Let me know if you need any more details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
