damondouglas commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1579833307
########## .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/beam-utility.apache-beam-testing.tfvars: ########## @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +project = "apache-beam-testing" +network = "default" +subnetwork = "default-f91f013bcf8bd369" +region = "us-central1" +router = "default-us-central1-router" +router_nat = "default-us-central1-router-nat" +cluster_name_override = "beam-utility" +cluster_name_prefix = "beam-utility" +service_account_id = "[email protected]" Review Comment: > In `google_service_account` datasource email is allowed https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/service_account#argument-reference Thank you for confirming and testing this. I recommend either changing the variable name to `service_account_email` and providing the email or `service_account_id` and changing the `tfvars` to be an id only. Personally, I prefer an ID since it means less data in the configuration but still works in the same project. ########## .test-infra/kafka/strimzi/README.md: ########## @@ -19,142 +19,31 @@ # Overview -This folder provisions a [strimzi.io](https://strimzi.io) kafka cluster on kubernetes. -Each folder is named in the required order of application. -These resources are managed using Kustomize. -See https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ -for more details on how Kustomize works with Kubernetes. +This module provisions the [strimzi operator](https://github.com/strimzi/strimzi-kafka-operator/tree/main/helm-charts/helm3/strimzi-kafka-operator). # Requirements +- [terraform](https://terraform.io) - Connection to a kubernetes cluster (See: [.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/google-kubernetes-engine](../../terraform/google-cloud-platform/google-kubernetes-engine) in this repository) - [kubectl](https://kubernetes.io/docs/reference/kubectl/) cli -# Preview kustomization - -To preview any of the kustomizations, run the following command. - -``` -kubectl kustomize <folder> -``` - -For example, to preview the kustomization for [01-strimzi-operator](01-strimzi-operator): - -``` -kubectl kustomize 01-strimzi-operator -``` - # Usage -Run the following commands provision the kafka cluster and assumes -the working directory [.test-infra/kafka-strimzi](.). - -**IMPORTANT: Each command needs to wait until the previous completes.** - -## 1. Install the strimzi operator. - -First install the strimzi operator: - -``` -kubectl apply -k 01-strimzi-operator -``` - -**IMPORTANT: Wait until completion before proceeding to the next step:** - -``` - kubectl get deploy strimzi-cluster-operator --namespace strimzi -w -``` - -## 3. Create the kafka cluster. - -A specific kafka cluster installation relies on kustomize overlays. - -### GKE internal load balanced kafka cluster - -The following command installs a Google Kubernetes Engine (GKE) -internally load balanced kafka cluster through the -[02-kafka-persistent/overlays/gke-internal-load-balanced](02-kafka-persistent/overlays/gke-internal-load-balanced) -overlay. - -``` -kubectl apply -k 02-kafka-persistent/overlays/gke-internal-load-balanced -``` - -You can watch while all the resources are created. - -``` -kubectl get all --namespace strimzi -``` - -#### Test Kafka connection on local machine -After all kafka cluster resources are created, you will want to validate the -Kafka instance running on the kubernetes cluster. - -In either of the commands you need to port forward the internal load -balancer service: - -``` -kubectl port-forward --namespace strimzi $(kubectl get pod --namespace strimzi --selector="strimzi.io/cluster=beam-testing-cluster,strimzi.io/kind=Kafka,strimzi.io/name=beam-testing-cluster-kafka" --output jsonpath='{.items[0].metadata.name}') 9094:9094 -``` - -##### Simple telnet - -In a new terminal after submitting the `kubectl port-forward` command above: - -``` -curl -v telnet://localhost:9094 -``` - -You should see: -``` -* Trying 127.0.0.1:9094... -* Connected to localhost (127.0.0.1) port 9094 (#0) -``` - -##### Use kcat - -See https://github.com/edenhill/kcat for instructions how to install `kcat`. +Simply follow standard terraform workflow to apply this module using the existing +[.test-infra/kafka/strimzi/common.tfvars](common.tfvars) for conventional defaults. -``` -kcat -L -b localhost:9094 -``` - -You should see something that looks like the following: -``` -Metadata for all topics (from broker -1: localhost:9094/bootstrap): - 3 brokers: - broker 0 at 10.128.0.12:9094 (controller) - broker 2 at 10.128.0.13:9094 - broker 1 at 10.128.0.11:9094 -``` - -#### Use with KafkaIO on Dataflow - -After all kafka cluster resources are created, you can run the -following command to find the kafka host and port. +Where: ``` -kubectl get svc beam-testing-cluster-kafka-external-bootstrap --namespace strimzi +DIR=.test-infra/kafka/strimzi Review Comment: Could we: 1) Move the terraform module into `01-strimzi-operator` folder? 2) Keeping .test-infra/kafka/strimzi/README.md where it is: change DIR=.test-infra/kafka/01-strimzi-operator ########## .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/beam-utility.apache-beam-testing.tfvars: ########## @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +project = "apache-beam-testing" +network = "default" +subnetwork = "default-f91f013bcf8bd369" +region = "us-central1" +router = "default-us-central1-router" +router_nat = "default-us-central1-router-nat" +cluster_name_override = "beam-utility" Review Comment: @volatilemolotov Thank you for listening. That would be great. ########## .test-infra/kafka/strimzi/02-kafka-persistent/README.md: ########## @@ -25,4 +25,19 @@ https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka The directories within [base](./base) contain the operator manifests and are named according to the version release. -See [.test-infra/kafka/strimzi](../README.md) for requirements and usage. +# Usage + +Simply deploy the cluster by using kustomize plugin of kubectl +``` +kubectl apply -k .test-infra/kafka/strimzi/02-kafka-persistent +``` +and wait until the cluster is deployed +``` +kubectl wait kafka beam-testing-cluster --for=condition=Ready Review Comment: I kept getting a timeout. I didn't have time to investigate this. Either investigate this or recommend using https://k9scli.io/ ########## .test-infra/kafka/strimzi/02-kafka-persistent/README.md: ########## @@ -25,4 +25,19 @@ https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka The directories within [base](./base) contain the operator manifests and are named according to the version release. -See [.test-infra/kafka/strimzi](../README.md) for requirements and usage. +# Usage + +Simply deploy the cluster by using kustomize plugin of kubectl +``` +kubectl apply -k .test-infra/kafka/strimzi/02-kafka-persistent Review Comment: I have two points: 1) When I tried this, I got the error: ``` error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '.test-infra/kafka/strimzi/02-kafka-persistent' ``` This worked: ``` kubectl apply -k .test-infra/kafka/strimzi/02-kafka-persistent/overlays/gke-internal-load-balanced ``` 2) Solution deployed into the default namespace. Was this intended? Original solution was in the default namespace. I don't mind either way. Following specifies the namespace. ``` kubectl apply -k .test-infra/kafka/strimzi/02-kafka-persistent/overlays/gke-internal-load-balanced --namespace=strimzi ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
