2019-07-11 07:45:37 UTC - Adam Varsano: Hi all, I start developing my business via openwhisk. I deployed the openwhisk on rancher via helm chart: <https://github.com/apache/incubator-openwhisk-deploy-kube>
The issue is that every few days/weeks something fall like zoo keeper pod. Then it doesn't come back to life, it goes into crash loop. I'm not feeling it is stable enough via kubernetes for production. Any ideas? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831137308500 ---- 2019-07-11 07:48:39 UTC - Dominic Kim: Since I am not working for IBM, I have no much idea but, AFAIK, IBM Cloud function has been operated on Kubernetes for server years. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831319309800 ---- 2019-07-11 07:50:36 UTC - Roberto Santiago: @Adam Varsano I have had the same issue as well. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831436310100 ---- 2019-07-11 07:55:06 UTC - chetanm: Can you open an issue for that and if possible add relevant log files. That may provide some clue https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831706312000 ---- 2019-07-11 07:55:43 UTC - Roberto Santiago: It seems like zookeeper pod can get evicted. When it comes back kafka is not able to find it. Then as a result the controller starts throwing errors. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831743312600 ---- 2019-07-11 07:57:13 UTC - chetanm: So far kube chart did not had pod disruption budgets. That may cause zookeeper quorum to break. Pod disruption budge support was added recently which may help to some extent https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562831833313800 ---- 2019-07-11 08:38:08 UTC - Sven Lange-Last: if you observe problems like pod eviction, check your kube resources with `kubectl -n <your_ns> describe pod ...`. you can do the same for your higher level resources like statefulsets, replicasets, etc. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834288315500 ---- 2019-07-11 08:38:26 UTC - Sven Lange-Last: pods are usually only evicted if something went wrong with your worker node. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834306315900 ---- 2019-07-11 08:38:39 UTC - Sven Lange-Last: maybe you are running too many components on a single worker node? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834319316300 ---- 2019-07-11 08:39:41 UTC - Sven Lange-Last: ibm cloud functions runs on ibm cloud kubernetes cluster. i can confirm that kube is stable enough to run openwhisk in production. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834381317100 ---- 2019-07-11 08:40:11 UTC - Sven Lange-Last: but i have to admit that we are not using <https://github.com/apache/incubator-openwhisk-deploy-kube> but use a homegrown deployment. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834411317600 ---- 2019-07-11 08:41:49 UTC - Roberto Santiago: @Sven Lange-Last I've thought about putting together my own as well. Would you be open to sharing some details about your deployment (and the thinking that went into how you are doing your deployment)? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834509318600 ---- 2019-07-11 08:42:44 UTC - Sven Lange-Last: i you have specific questions, i may be able to share some details. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834564319200 ---- 2019-07-11 08:44:45 UTC - Roberto Santiago: The two areas I have recurrently had issues with is kafka-zookeeper (similar to @Adam Varsano) and couchdb. What are you doing around deployment with those two pieces? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834685320500 ---- 2019-07-11 08:45:41 UTC - Sven Lange-Last: we are using ibm cloudant db service instead of running our own db. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834741321400 ---- 2019-07-11 08:45:57 UTC - Adam Varsano: So what is the best practice for deploy openwhisk with the most stable way? Can you explain more about your homegrown deployment? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834757321800 ---- 2019-07-11 08:47:30 UTC - Sven Lange-Last: regarding zookeeper / kafka. we made sure that these run on their own (large) worker nodes and have enough cpu / disk resources. these two have persistent volume claims to make sure they cannot loose their data. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834850322900 ---- 2019-07-11 08:48:38 UTC - Sven Lange-Last: and for prod envs, it’s better to set up zookeeper / kafka in an ha setup with replication. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834918323800 ---- 2019-07-11 08:49:02 UTC - Adam Varsano: So we should ignore the helm chart? deploy it manually? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562834942324200 ---- 2019-07-11 08:50:36 UTC - Sven Lange-Last: to be honest: i don’t know the helm charts in detail. it’s better to discuss with @Dave Grove. but running on kube requires that you somehow define your deployment in terms of replicasets, statefulsets, and so on. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835036325400 ---- 2019-07-11 08:51:14 UTC - Sven Lange-Last: that said - what does manual deployment mean on kube? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835074326000 ---- 2019-07-11 08:54:29 UTC - Adam Varsano: Today the helm chart automaticly install zookeeper, kafka, invoker, couchdb, init-couchdb, kafka-provider..... it also connect them together. Manually I mean by install by myself kafka, zookeeper,... and connect them by myself, sound to me like not easy way https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835269327800 ---- 2019-07-11 08:55:33 UTC - Sven Lange-Last: do you just want to try openwhisk on kube? or is your goal to set up a prod env? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835333328500 ---- 2019-07-11 08:55:56 UTC - chetanm: The default value in kube repo are for minimal setup suitable for local development. Some values would need to be adapted to make it suitable for prod run https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835356328900 ---- 2019-07-11 08:56:33 UTC - Adam Varsano: I want prod env with millions of users https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835393329200 ---- 2019-07-11 08:56:40 UTC - Adam Varsano: it must be super stable https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835400329400 ---- 2019-07-11 08:57:10 UTC - Sven Lange-Last: which container factory do you plan to use? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835430329700 ---- 2019-07-11 08:57:36 UTC - Adam Varsano: kubernetes, via rancher https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835456329900 ---- 2019-07-11 08:58:14 UTC - Adam Varsano: container factory = kubernetes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835494330500 ---- 2019-07-11 08:59:12 UTC - Sven Lange-Last: well, the kube container factory is not really fast compared to the docker container factory. you should perform thorough testing and make a decision whether you want to serve millions of users with this container factory… https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835552331500 ---- 2019-07-11 09:01:33 UTC - Adam Varsano: The problem with docker container factory is that it make the invoker to be DaemonSet It means one pod per node https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835693332000 ---- 2019-07-11 09:01:51 UTC - Adam Varsano: I'm taking when installing the helm chart https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835711332400 ---- 2019-07-11 09:02:17 UTC - Sven Lange-Last: what’s the problem with that approach? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835737332700 ---- 2019-07-11 09:02:48 UTC - Adam Varsano: When lots of request are comming only 1 container serving the requests instead of duplicate to more containers https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835768333300 ---- 2019-07-11 09:06:16 UTC - Sven Lange-Last: hm, each invoker pod manages multiple containers - not only a single container. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562835976333800 ---- 2019-07-11 09:08:43 UTC - Adam Varsano: ok, this is not the main issue, the main issue is that every few days zookeeper is going into crash loop and the all env is down. How can I solve this? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562836123334800 ---- 2019-07-11 09:11:43 UTC - Sven Lange-Last: my advice is that you thoroughly check zookeeper logs and the kube pods running zookeeper. use `kubectl describe` to understand what’s going on with the zookeeper pod(s). https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562836303336000 ---- 2019-07-11 09:12:31 UTC - chetanm: How many instances of zookeeper are you running? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562836351336400 ---- 2019-07-11 09:12:59 UTC - chetanm: Query was for adamva https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562836379336700 ---- 2019-07-11 11:10:12 UTC - Rodric Rabbah: @James Thomas did you have real time talk transcription?!? whoa… was it serverless? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562843412336800?thread_ts=1562750641.298700&cid=C3TPCAQG1 ---- 2019-07-11 13:18:19 UTC - Dave Grove: @Adam Varsano for a production setting, you definitely need to change the default value of `replicaCount` for `zookeeper`, `kafka`, `controller`, `invoker`, etc. to be `>1`. As @Sven Lange-Last said, you also should run an external CouchDB (not coupled to the helm chart). The chart supports this. This allows you to manage the lifecycle of the database separately from the core OpenWhisk. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562851099340200 ---- 2019-07-11 14:49:10 UTC - James Thomas: bah - no livestream today I’m afraid! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562856550345000?thread_ts=1562750641.298700&cid=C3TPCAQG1 ---- 2019-07-11 14:49:41 UTC - James Thomas: lots of interesting web assembly stuff at the conference - got me thinking about a webasm runtime in ow! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562856581345200?thread_ts=1562750641.298700&cid=C3TPCAQG1 ---- 2019-07-11 14:50:09 UTC - Rodric Rabbah: ;) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562856609345500?thread_ts=1562750641.298700&cid=C3TPCAQG1 ---- 2019-07-11 15:01:34 UTC - Michael Schmidt: So I think the issues here revolves around the fact that my kafka is not exposed outside the cluster. I really want wsk to create the trigger from inside the cluster though and do its checks from inside the cluster. To work around this, I was wondering if there was a wsk container that I could actually launch into k8s for testing purposes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562857294346000?thread_ts=1562671501.292600&cid=C3TPCAQG1 ---- 2019-07-11 17:27:35 UTC - Michael Schmidt: I am attempting an offiline install of OpenWhisk on k8s with helm and the couchdb fails because it cannot reach github, what is the configuration variable needed to point it to an internal git project? Or can I turn off the call to github? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562866055347400 ---- 2019-07-11 17:42:40 UTC - Michael Schmidt: logs from couchdb ``` Cloning into '/openwhish' ... fatal: unable to access '<https://github.com/apache/incubator-openwhisk/>': Could not resolve host: <http://github.com|github.com> ``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562866960349200 ---- 2019-07-11 17:42:45 UTC - Michael Schmidt: the init script https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562866965349400 ---- 2019-07-11 18:03:03 UTC - Dave Grove: you can try editing <https://github.com/apache/incubator-openwhisk-deploy-kube/blob/master/helm/openwhisk/configMapFiles/initCouchDB/initdb.sh> to change where you piull from. It’s unfortunately hardwired in the script. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868183350100 ---- 2019-07-11 18:03:22 UTC - Dave Grove: arguably should/could be made configurable https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868202350600 ---- 2019-07-11 18:03:37 UTC - Michael Schmidt: Okay so then do I need to build that container in my closed env? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868217350900 ---- 2019-07-11 18:04:30 UTC - Dave Grove: that file gets mounted as a configmap when you do the helm install. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868270351500 ---- 2019-07-11 18:04:42 UTC - Dave Grove: should work to edit the file locally & then deploy with helm https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868282351900 ---- 2019-07-11 18:04:44 UTC - Michael Schmidt: oh https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868284352100 ---- 2019-07-11 18:04:48 UTC - Michael Schmidt: that isn't that bad... https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868288352300 ---- 2019-07-11 18:04:51 UTC - Michael Schmidt: thank you! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562868291352500 ---- 2019-07-11 19:25:22 UTC - didier amyot: Hi guys, Quick question, is it possible to run OpenWhisk without using docker. I deploy software in restrictive environments and docker is not an option for us. We usually deploy on centos 7 machines without root access. In the best world, we would like to be able to deploy java (jvm) and python actions. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873122352900 ---- 2019-07-11 19:26:48 UTC - Michael Schmidt: It is https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873208353100 ---- 2019-07-11 19:26:52 UTC - Michael Schmidt: let me get you the link https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873212353300 ---- 2019-07-11 19:27:10 UTC - Michael Schmidt: oh wait without docker... https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873230353700 ---- 2019-07-11 19:27:11 UTC - Michael Schmidt: mmmmm https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873231353900 ---- 2019-07-11 19:27:13 UTC - Michael Schmidt: let me see https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873233354100 ---- 2019-07-11 19:28:21 UTC - didier amyot: i suppose docker is a building block of open whisk https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873301355000 ---- 2019-07-11 19:28:29 UTC - Michael Schmidt: <https://github.com/apache/incubator-openwhisk-devtools/tree/master/node-local> <https://github.com/apache/incubator-openwhisk/tree/master/core/standalone> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873309355200 ---- 2019-07-11 19:28:43 UTC - Michael Schmidt: Its usually hard to deploy things without some access to do so :stuck_out_tongue: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873323355600 ---- 2019-07-11 19:28:59 UTC - Michael Schmidt: That stand alone server in java may save you https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873339355900 ---- 2019-07-11 19:29:38 UTC - didier amyot: to be fair, I am a humble programmer, and I work with geniuses devops guys https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873378357100 ---- 2019-07-11 19:29:46 UTC - Michael Schmidt: Thanks to @chetanm for bringing this link up a few days ago. credit to him :slightly_smiling_face: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873386357300 ---- 2019-07-11 19:30:05 UTC - Michael Schmidt: as a devops guy, not geniuses... https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873405357600 ---- 2019-07-11 19:30:32 UTC - Michael Schmidt: shouldn't they be setting this up for you then! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873432358200 ---- 2019-07-11 19:30:55 UTC - didier amyot: open whisk will not need to run the actions inside some kind of containers? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873455358700 ---- 2019-07-11 19:33:09 UTC - didier amyot: According to the standalone README, > It only needs Docker and Java to for running. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873589359100 ---- 2019-07-11 19:33:26 UTC - didier amyot: my problem is that in production I will not have access to docker https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873606359700 ---- 2019-07-11 19:34:14 UTC - Michael Schmidt: Does your system run docker swarm mesos or k8s? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873654360800 ---- 2019-07-11 19:34:31 UTC - didier amyot: mesos but without docker https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873671361400 ---- 2019-07-11 19:34:42 UTC - Michael Schmidt: I didn't even know that was a thing :stuck_out_tongue: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873682362000 ---- 2019-07-11 19:35:03 UTC - Michael Schmidt: <https://stackoverflow.com/questions/36510813/run-java-app-on-mesos-without-docker> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873703362400 ---- 2019-07-11 19:35:05 UTC - Michael Schmidt: huh it is https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873705362900 ---- 2019-07-11 19:35:45 UTC - didier amyot: so, theoritically, we could use openwhisk with mesos, without docker https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873745363800 ---- 2019-07-11 19:35:58 UTC - Michael Schmidt: I THINK (I am not the creator of openwhisk) it could be possible? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873758364500 ---- 2019-07-11 19:36:14 UTC - didier amyot: rock on, I have a weekend project! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873774365000 ---- 2019-07-11 19:36:14 UTC - Michael Schmidt: yah agreed https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873774365100 ---- 2019-07-11 19:36:21 UTC - Michael Schmidt: idk https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873781365300 ---- 2019-07-11 19:36:26 UTC - Michael Schmidt: if it can that would be cool https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873786365500 ---- 2019-07-11 19:36:54 UTC - Michael Schmidt: So it will depend on if there is "container factory" for mesos https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873814366300 ---- 2019-07-11 19:37:06 UTC - didier amyot: because I am not the only one living in a corporate world where people are scared of docker https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873826366800 ---- 2019-07-11 19:37:10 UTC - Michael Schmidt: because that will determine if you can launch native mesos stuff or not https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873830367000 ---- 2019-07-11 19:37:24 UTC - Michael Schmidt: generally openwhisk default is to run its docker containers in the background https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873844367400 ---- 2019-07-11 19:37:31 UTC - Michael Schmidt: outside of K8s mesos and docker swarm https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873851367700 ---- 2019-07-11 19:37:45 UTC - Michael Schmidt: soooo, there is a good chance this is all container based techg https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873865368100 ---- 2019-07-11 19:37:52 UTC - Michael Schmidt: tech* https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873872368300 ---- 2019-07-11 19:39:03 UTC - didier amyot: But I least not I know it worth to take a deepdive into openwhisk. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873943369300 ---- 2019-07-11 19:39:36 UTC - Michael Schmidt: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873976369600 ---- 2019-07-11 19:39:41 UTC - Michael Schmidt: This makes me think its all containers https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873981370000 ---- 2019-07-11 19:39:49 UTC - Michael Schmidt: yeah, I'd at least dive into it https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562873989370300 ---- 2019-07-11 19:40:05 UTC - Michael Schmidt: maybe you all decide to install docker on all machines outside your current mesos install https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562874005370700 ---- 2019-07-11 19:40:23 UTC - Michael Schmidt: and then mesos has openwhisk, but in the background openwhisk maintains its own container set https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562874023371300 ---- 2019-07-11 19:41:12 UTC - Dave Grove: There is a MesosContainerFactory. <https://github.com/apache/incubator-openwhisk/tree/master/common/scala/src/main/scala/org/apache/openwhisk/core/mesos> partyparrot : Michael Schmidt +1 : didier amyot https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562874072371800 ---- 2019-07-11 19:41:51 UTC - Dave Grove: Haven’t used it. but maybe @Tyson Norris can comment. Adobe is the main driver for OpenWhisk on Mesos. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562874111372700 ---- 2019-07-11 23:17:26 UTC - Dragos Dascalita Haut: Yes, Adobe is using the MesosContainerFactory, but with Docker. See <https://github.com/adobe-apiplatform/mesos-actor/blob/master/src/main/scala/com/adobe/api/platform/runtime/mesos/TaskBuilder.scala#L70> . In theory one _could_ run Mesos tasks w/o Docker; do you plan to use Kata W Firecracker instead ? or what's the plan to execute the code ? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562887046375000 ----
