tamis-laan opened a new issue, #30724: URL: https://github.com/apache/beam/issues/30724
I'm trying out Apache Beam for the first time and I'm interested in running it in k8s. As far as I understand the best way to do this is to use Apache Flink on k8s as the run environment. I followed this guide to setup Apache Flink in session mode: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions ``` bash # Configuration and service definition kubectl create -f flink-configuration-configmap.yaml kubectl create -f jobmanager-service.yaml # Create the deployments for the cluster kubectl create -f jobmanager-session-deployment-non-ha.yaml kubectl create -f taskmanager-session-deployment.yaml ``` This works and creates a jobmanager with ports: 6123 (rpc) 6124 (blob-server) 8081 (webui) It also creates two taskmanagers with ports: 6122 As far as I understand jobs are submitted to the jobmanager on port 8081 and the taskmanagers handle the job execution. I'm using the following go code as an example: https://raw.githubusercontent.com/apache/beam/master/sdks/go/examples/wordcount/wordcount.go I'm forwarding jobmanager:8081 in kubernetes to localhost:8081 and executing the go code as followed: ``` bash go run main.go --runner=PortableRunner --endpoint=localhost:8081 --environment_type=LOOPBACK ``` This results in a failure to connect: ``` 2024/03/23 11:49:34 stopping Loopback, and 0 workers 2024/03/23 11:49:34 Failed to execute job: connecting to job service failed to dial server at localhost:8081 caused by: context deadline exceeded exit status 1 ``` Because I'm new to Beam and Flink I'm not sure where I am going wrong. Can anyone with some more experience give me a pointer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
