2019-08-10 11:26:19 UTC - Michael Schmidt: Hi so I now have had time to come back to the kafka trigger in openwhisk, Some basics first: I run openwhisk on a k8s cluster and successfully deploy with helm (which is awesome) I use the k8s factory because I'd like to run wsk triggers that launch based on watching a kafka queue that is running only exposed inside the cluster Right now, my kafka cluster runs in k8s, but it runs in a node port configuration where all brokers are using a node port Ultimately my goal is to leverage the serverless/serverless libraries to allow people to launch to several cloud offerings for serverless workloads and allow them to data crunch off kafka queues that fluxuate in flow rate
So now questions: Is triggers in openwhisk the appriate way to allow for auto scaling of functions in openwhisk (forgetting the serverless/serverless lib for a second) If I am successful in implimenting a trigger in wsk (so far haven't had luck yet) off a kafka queue is this even doable if I abstract up to the serverless/serverless library? Has anyone had luck running OpenWhisk on Kubernetes and reading off a kafka queueue? If so do you happen to have any docs? I was able to successfully produce a message to kafka from the wsk cli. I say "from the wsk" because when I had a kafka cluster that was running solely in the k8s cluster I run As I run into issue I can post some logs, but haven't gotten a chance to do this in a few weeks so I need to play around first. I feel close since I can produce to a queue If anyone has had success your experience and pain points would be good to know as I try to solve this out :slightly_smiling_face: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1565436379477900 ----
