Hi Brandon, Thanks for the response. Kafka is publishing a message to a topic in two-step process: a. Connect to a broker which is a pod running on a kubernetes worker and get metadata about the topic i.e. who is the leader of the topics partitions (info stored in zookeeper) b. Connect to that leader and start sending data to it
Say I would use port forwarding to solve phase a. However, for phase b I would need to have some port opened on the worker node where the kafka leader container is running. Since I want my publisher to connect directly to the worker node running the kafka leader. On top of that I see that the portforward.go mentions something about a "upgraded HTTP request" for the communication. I want to be able to connect on the TCP level to the kafka node. One approach from my side was to use nodePort to open a port on the nodes where kafka is running. However, this seems not to work properly see https://github.com/kubernetes/kubernetes/issues/23920. Thanks, George On Tuesday, June 6, 2017 at 2:24:19 AM UTC+2, Brandon Philips wrote: > Find a Kubernetes Pod by Label Selector and Port-forward Locally > > > Given a namespace "your-namespace" and a label query that identifies the pods > you are interested in connect to a particular pod instance. If the label > selector doesn't find a unique pod it will connect to the first pod by name. > Ensure you replace 8080 with your pod's port. > > > kubectl -n your-namespace get pods -n your-namespace -l run=hello-world -o > jsonpath='{.items[1].metadata.name}' | xargs -I{} kubectl -n your-namespace > port-forward {} 8080 > > > On Wed, May 31, 2017 at 1:43 AM <george...@gmail.com> wrote: > Hello everybody, > > > > I've setup a 3 node kafka cluster running on kubernetes using stateful sets > and headless services. > > The problem is now I want read/write data from/to the cluster from outside > the cluster. > > I have found a couple of suggestions here > https://stackoverflow.com/questions/41868161/kafka-in-kubernetes-cluster-how-to-publish-consume-messages-from-outside-of-kub. > > > > Essentially, I've found 2 options: > > 1. Use hostNetwork: true when starting the kafka container and use as > ADVERTISED_HOST in kafka the kubernetes worker node - this has > > 2. Use Services type NodePort to expose the kafka broker port to the > outside world - drawback here is that it will open a port on all worker nodes > and second the request will go through the kube proxy > > > > On top of this I still need to sync my external dns system with the dns > of the kafka worker nodes. > > > > Could anyone share their experience on how they achieved this? > > > > Thanks, > > George > > > > -- > > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-use...@googlegroups.com. > > To post to this group, send email to kubernet...@googlegroups.com. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
