2020-03-19 13:05:44 UTC - Mike Ludwig: Has anyone seen the invoker consistently run out of memory after a few hours? I think this is happening because timeouts aren't being enforced. More details here: <https://stackoverflow.com/questions/60723597/openwhisk-out-of-function-memory-not-enforcing-function-timeout> Installed openwhisk on k3s using helm. https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584623144010700?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:02:46 UTC - Keerthi Kumar S R: @Dave Grove is there a way to pass pod annotation to the action pod https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584626566011200?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 14:02:58 UTC - Keerthi Kumar S R: In our infra istio is used. I want to disable istio for the action pod... Which can be done by passing annotation... https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584626578011400?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 14:03:29 UTC - Keerthi Kumar S R: is there a way to do this?? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584626609011600?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 14:08:38 UTC - Dave Grove: There is support for customizing the template used for action pods. <https://github.com/apache/openwhisk/blob/master/core/invoker/src/main/resources/application.conf#L80> . https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584626918011800?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 14:13:32 UTC - Rodric Rabbah: hi Mike https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584627212012100?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:14:19 UTC - Rodric Rabbah: function containers should get reclaimed after an idle period. https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584627259012300?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:15:38 UTC - Rodric Rabbah: an function timeout means something else though - this is the max duration of an individual activation. Its container may stick around longer so it can be reused (this becomes a warm start). https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584627338012500?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:15:52 UTC - Rodric Rabbah: are you saying/seeing that the max duration is also not being enforced? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584627352012700?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:21:02 UTC - Mike Ludwig: Hi Rodric, it may just be that the pods are sticking around. Do the warm pods still use up function memory? And if so is there a way to limit the number of warm pods? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584627662012900?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:35:11 UTC - Mike Ludwig: Perhaps lowering `actionsInvokesConcurrent` here will do this <https://github.com/apache/openwhisk-deploy-kube/blob/master/helm/openwhisk/values.yaml#L73> https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584628511013100?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:53:56 UTC - Rodric Rabbah: @Dave Grove i think is the better authority on this https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584629636013300?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:54:19 UTC - Rodric Rabbah: action invokes concurrent limits number of activations that the system can handle at once https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584629659013500?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 14:54:43 UTC - Rodric Rabbah: i assume you’re using the kubernetes container factory - is that right? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584629683013700?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 15:00:24 UTC - Dave Grove: warm pods count against the function memory. However, the invoker is supposed to terminate warm pods that aren’t actually running anything when it is under memory pressure to make room for things it is supposed to be running. https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584630024014000?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 15:08:20 UTC - Mike Ludwig: Yes the invoker containerFactory settings are all set to default. One other thing to note is I'm using docker as the container runtime, maybe containerd behaves better? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584630500014200?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 15:27:20 UTC - Dave Grove: the KubernetesContainerFactory doesn’t make assumptions about docker vs. containerd. So it shouldn’t matter https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584631640014400?thread_ts=1584623144.010700&cid=C4J3R7JFL ---- 2020-03-19 17:09:57 UTC - Keerthi Kumar S R: @Dave Grove i think is some silly mistake... i am actually not able to pass multiline yaml template.. this is how i am try to pass... can you help ?? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584637797014600?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 17:11:32 UTC - Keerthi Kumar S R: i am actually getting some error as https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584637892015000?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 17:11:34 UTC - Keerthi Kumar S R: expected <block end>, but found '<scalar>' in 'reader', line 1, column 3: """--- apiVersion: "v1" kind: "Pod ... https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584637894015200?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 17:45:44 UTC - Dave Grove: @chetanm or @Neeraj Mangal — does either of you have an example of how to customize the action pod template? https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584639944015400?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 17:57:08 UTC - Neeraj Mangal: Hi @Keerthi Kumar S R This is the old conversation we had related to this, which may help. <https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580790184079000?thread_ts=1580760913.071500&cid=C3TPCAQG1> Idea it to add a file as the pod template, add it as CM and mount it. Please let me know if this does not work out. https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584640628015800?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 18:26:56 UTC - Keerthi Kumar S R: @Dave Grove thank you... even i tried with volume mount... its working https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584642416016700?thread_ts=1584626566.011200&cid=C4J3R7JFL ---- 2020-03-19 20:56:56 UTC - Hernan Moreno: hello.. How could I deploy openwhisk if I'm using kubernetes with k3d(k3s in docker) https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584651416017600?thread_ts=1584651416.017600&cid=C4J3R7JFL ---- 2020-03-19 21:35:45 UTC - Dave Grove: our automated testing uses kind. have not tried k3d. most likely the helm chart will deploy more or less fine, but you’ll need to figure out ingress for k3d. that’s where most of the variation is between different flavors of kubernetes https://openwhisk-team.slack.com/archives/C4J3R7JFL/p1584653745017800?thread_ts=1584651416.017600&cid=C4J3R7JFL ----