volatilemolotov commented on code in PR #30847: URL: https://github.com/apache/beam/pull/30847#discussion_r1580929019
########## .test-infra/kafka/strimzi/README.md: ########## @@ -19,142 +19,31 @@ # Overview -This folder provisions a [strimzi.io](https://strimzi.io) kafka cluster on kubernetes. -Each folder is named in the required order of application. -These resources are managed using Kustomize. -See https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ -for more details on how Kustomize works with Kubernetes. +This module provisions the [strimzi operator](https://github.com/strimzi/strimzi-kafka-operator/tree/main/helm-charts/helm3/strimzi-kafka-operator). # Requirements +- [terraform](https://terraform.io) - Connection to a kubernetes cluster (See: [.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/google-kubernetes-engine](../../terraform/google-cloud-platform/google-kubernetes-engine) in this repository) - [kubectl](https://kubernetes.io/docs/reference/kubectl/) cli -# Preview kustomization - -To preview any of the kustomizations, run the following command. - -``` -kubectl kustomize <folder> -``` - -For example, to preview the kustomization for [01-strimzi-operator](01-strimzi-operator): - -``` -kubectl kustomize 01-strimzi-operator -``` - # Usage -Run the following commands provision the kafka cluster and assumes -the working directory [.test-infra/kafka-strimzi](.). - -**IMPORTANT: Each command needs to wait until the previous completes.** - -## 1. Install the strimzi operator. - -First install the strimzi operator: - -``` -kubectl apply -k 01-strimzi-operator -``` - -**IMPORTANT: Wait until completion before proceeding to the next step:** - -``` - kubectl get deploy strimzi-cluster-operator --namespace strimzi -w -``` - -## 3. Create the kafka cluster. - -A specific kafka cluster installation relies on kustomize overlays. - -### GKE internal load balanced kafka cluster - -The following command installs a Google Kubernetes Engine (GKE) -internally load balanced kafka cluster through the -[02-kafka-persistent/overlays/gke-internal-load-balanced](02-kafka-persistent/overlays/gke-internal-load-balanced) -overlay. - -``` -kubectl apply -k 02-kafka-persistent/overlays/gke-internal-load-balanced -``` - -You can watch while all the resources are created. - -``` -kubectl get all --namespace strimzi -``` - -#### Test Kafka connection on local machine -After all kafka cluster resources are created, you will want to validate the -Kafka instance running on the kubernetes cluster. - -In either of the commands you need to port forward the internal load -balancer service: - -``` -kubectl port-forward --namespace strimzi $(kubectl get pod --namespace strimzi --selector="strimzi.io/cluster=beam-testing-cluster,strimzi.io/kind=Kafka,strimzi.io/name=beam-testing-cluster-kafka" --output jsonpath='{.items[0].metadata.name}') 9094:9094 -``` - -##### Simple telnet - -In a new terminal after submitting the `kubectl port-forward` command above: - -``` -curl -v telnet://localhost:9094 -``` - -You should see: -``` -* Trying 127.0.0.1:9094... -* Connected to localhost (127.0.0.1) port 9094 (#0) -``` - -##### Use kcat - -See https://github.com/edenhill/kcat for instructions how to install `kcat`. +Simply follow standard terraform workflow to apply this module using the existing +[.test-infra/kafka/strimzi/common.tfvars](common.tfvars) for conventional defaults. -``` -kcat -L -b localhost:9094 -``` - -You should see something that looks like the following: -``` -Metadata for all topics (from broker -1: localhost:9094/bootstrap): - 3 brokers: - broker 0 at 10.128.0.12:9094 (controller) - broker 2 at 10.128.0.13:9094 - broker 1 at 10.128.0.11:9094 -``` - -#### Use with KafkaIO on Dataflow - -After all kafka cluster resources are created, you can run the -following command to find the kafka host and port. +Where: ``` -kubectl get svc beam-testing-cluster-kafka-external-bootstrap --namespace strimzi +DIR=.test-infra/kafka/strimzi Review Comment: Also left the README.md in the strimzi folder and updated the DIR instruction -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
