2020-03-06 01:46:56 UTC - Dominic Kim: It is briefly mentioned here: <https://github.com/apache/openwhisk-deploy-kube/blob/caac4d6de2193301c948b4091345224fc69fdbce/docs/configurationChoices.md#using-an-external-database> but I am not aware of any document. I would defer to IBM guys who are operating OW on K8S. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583459216204900?thread_ts=1583279584.123800&cid=C3TPCAQG1 ---- 2020-03-06 02:32:32 UTC - Jiang PengCheng: the `CONFIG_whisk_activationStore_elasticsearch_hosts` should not contain the `http://` or `https://` , just use ip address here, e.g. `192.168.1.1:9200,192.168.1.2:9200` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583461952205100?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 02:33:01 UTC - Jiang PengCheng: and if your ES doesn't need any auth, you can provide any username/password https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583461981205300?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 02:34:28 UTC - Jiang PengCheng: I'm not familar with `openwhisk-deloy-kube repo` , it would be great if you can work it out and make a PR +1 : Bilal https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583462068205500?thread_ts=1583430426.197100&cid=C3TPCAQG1 ---- 2020-03-06 04:58:33 UTC - chetanm: @Brendan Doyle You mean passing config via `-c` option? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583470713205700?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:02:36 UTC - Brendan Doyle: In the jar there’s just normal command line options, -m for example for the manifest. How do I set that when running the docker container, I don’t see any documentation on it because there’s just the run script for docker which doesn’t pass in any configs https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583470956208200?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:05:25 UTC - chetanm: With Docker passing external files would be tricky. You would need to mount the local directory as volume and then refer to mounted path https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471125208400?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:05:46 UTC - chetanm: The start scripts does pass the script args to Docker command <https://github.com/apache/openwhisk/blob/master/core/standalone/start.sh> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471146208600?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:06:12 UTC - chetanm: But then you would need to craft the docker command manually to pass the volume settings https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471172208900?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:06:35 UTC - chetanm: For such cases I believe using the jar mode is better and convinient https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471195209100?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:06:43 UTC - chetanm: @Michele Sciabarra Thoughts? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471203209300?thread_ts=1583439011.202600&cid=C3TPCAQG1 ---- 2020-03-06 05:38:50 UTC - Carlos Santana: out of `user-memory` is not indication that invokers are unhealthy just bussy, the activations would just seat on kafka topic waiting, from what I understand https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583473130210300?thread_ts=1583436410.201700&cid=C3TPCAQG1 ---- 2020-03-06 11:16:00 UTC - Tom Barber: I left OW alone for 24 hours and now it does this: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493360210700 ---- 2020-03-06 11:16:02 UTC - Tom Barber: ```MT-109900:~ barber$ wsk action delete /guest/pixlize-serverless-dev-triggerPiquantJob error: Unable to delete action 'pixlize-serverless-dev-triggerPiquantJob': Unexpected http response code: 500 Internal Server Error (details: {"error":"unknown_error","reason":"undefined"} ) (code VXdgUj5Of9881NKLgHQy3Tvb4neBxZEd) Run 'wsk --help' for usage. MT-109900:~ barber$ wsk action delete /guest/pixlize-serverless-dev-testParams error: Unable to delete action 'pixlize-serverless-dev-testParams': Unexpected http response code: 500 Internal Server Error (details: {"error":"unknown_error","reason":"undefined"} ) (code kQ0bBAPLTI4nCTIndcQdM2wVr2jq1e29) Run 'wsk --help' for usage.``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493362211000 ---- 2020-03-06 11:16:14 UTC - Tom Barber: any guesses https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493374211200 ---- 2020-03-06 11:18:10 UTC - Tom Barber: oh a crap load of Couchdb errors https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493490211500 ---- 2020-03-06 11:18:17 UTC - Tom Barber: :exploding_head: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493497211800 ---- 2020-03-06 11:18:51 UTC - Tom Barber: always couchdb that fails in some way or other https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493531212100 ---- 2020-03-06 11:21:21 UTC - Tom Barber: and then I restart couchdb and its completely hosed the install https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493681212500 ---- 2020-03-06 11:22:23 UTC - Tom Barber: thats even with a PVC in place though, I figured my couchdb woes would go away with some persistence https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493743213000 ---- 2020-03-06 11:23:26 UTC - Tom Barber: oh... the 2GB default volume size is full up https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493806213300 ---- 2020-03-06 11:23:29 UTC - Tom Barber: sad times https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493809213500 ---- 2020-03-06 11:31:10 UTC - Tom Barber: alrighty https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583494270213800 ---- 2020-03-06 11:31:27 UTC - Tom Barber: hopelly 50Gi will last it a little longer than 2Gi :slightly_smiling_face: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583494287214300?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 11:45:23 UTC - Rodric Rabbah: Are you filling up with activations? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583495123214700?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:12:15 UTC - Bilal: I also had some issues with volume size on kube, I think it was for Kafka? Might have been the DB as well. I bumped up most of the internals to around that level, @Tom Barber, after talking to @Rodric Rabbah about it. There is a retention policy set per deployment that affects how long things are retained, but I haven't messed around with that as I don't need long term retention.
I had about 30,000 activations over the last 24 hours. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507535214900?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:17:17 UTC - Bilal: "Decoupling the database increases operational flexibility, for example by enabling blue/green deployments of OpenWhisk using a *shared database instance*." (emphasis added) Aha, missed that bit when I read over this page a while ago. So I'd have two sets of openwhisk pointing at the same DB, then I can upgrade one OW deployment and if all is well make that live. Shouldn't be too hard, I already have a loadbalancer in front of everything pointing at my OW deployment, so it sounds like all I have to do is change where the loadbalancer points. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507837215200?thread_ts=1583279584.123800&cid=C3TPCAQG1 ---- 2020-03-06 15:17:50 UTC - Bilal: Well, I say "all I have to do" and make it sound simple, but there's always issues that pop up, lol https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507870215400?thread_ts=1583279584.123800&cid=C3TPCAQG1 ---- 2020-03-06 15:19:26 UTC - Bilal: Thanks! I also found out that `options` takes strings, and `CONFIG_WHISK` should be replaced with `-Dwhisk` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507966215600?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 15:19:40 UTC - Bilal: Still messing around with it though https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507980215800?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 15:21:30 UTC - Tom Barber: our stuff is minimal in usage currently just internal testing so nothing like @Bilal https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508090216100?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:22:11 UTC - Tom Barber: I blew it away and stuck 50Gb in, so I guess it wont occur again for now. Thanks for the pointers @Bilal I'll check the other volumes also to ensure I don't blow through them https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508131216300?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:25:04 UTC - Bilal: Well I'm still in internal testing phase too :slightly_smiling_face: I just had my OW listen to a bunch of stuff as a way prove stability while I continue to work on more stuff for my OW deployment. I think you are customizing your OW a lot more than me, but from what I've read sounds like you're working on a super cool project for NASA. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508304216500?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:33:35 UTC - Rodric Rabbah: <https://youtu.be/X2UaFuJsqxk> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508815216700?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 15:38:18 UTC - Bilal: Absolutely fantastic, thanks for sharing. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583509098217000?thread_ts=1583494287.214300&cid=C3TPCAQG1 ---- 2020-03-06 16:11:31 UTC - Bilal: ``` WHISK_LOGS_DIR: CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'owdev-whisk.config'> Optional: false CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'owdev-db.auth'> Optional: false CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'owdev-db.auth'> Optional: false CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_host: owdev-couchdb.openwhisk.svc.cluster.local CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'owdev-db.config'> Optional: false``` More messing around/thinking outloud. So I think `CONFIG_whisk_couchdb_databases_WhiskActivation:` needs to point to the elastic search whereas the `WhiskEntitiy` and `WhiskAuth` will remain in couchDB https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511091217200?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 16:12:14 UTC - Bilal: and I do see my invoker options being set ```INVOKER_OPTS: -Dwhisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider -Dwhisk_activationStore_elasticsearch_protocol=http -Dwhisk_activationStore_elasticsearch_hosts=172.18.0.1:9200 -Dwhisk_activationStore_elasticsearch_indexPattern=openwhisk -Dwhisk_activationStore_elasticsearch_username=admin -Dwhisk_activationStore_elasticsearch_password=admin -Dkubernetes.master=https://$KUBERNETES_SERVICE_HOST -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511134217400?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 16:24:40 UTC - Bilal: Oh oh oh, the `openwhisk-deploy-kube` repo pulls ansible playbooks from the main `openwhisk` repo when the CouchDB is initialized https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511880217600?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 17:46:34 UTC - Bilal: @Dave Grove Any advice or direction on how to use <https://github.com/apache/openwhisk/pull/4724> with the openwhisk-deploy-kube/helm?. Unfortunately, I've never used ansible before. I feel like I understand what needs to happen but I can't wrap my head around how. I've poked around in the changes made in the PR and from what I understand, the CouchDB gets init and created using the same ansible playbook in both the helm/kube deployment and the ansible deployment. Where it diverges is the nextstep in the readme from the ElasticSearch PR `ansible-playbook -i environments/<environment> openwhisk.yml -e db_activation_backend=ElasticSearch` The backend gets set to ElasticSearch in `ansible/group_vars/all` and the only thing I can see that interacts with this setting are the `deploy.yml` playbooks for both the invoker and controller which sets the options above that Jiang PengCheng helped me out with. I'm a bit stuck on what to try, any help is appreciated, thanks! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583516794217800?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 18:24:56 UTC - Brendan Doyle: quick question. when we do a get on an activation for the result through the http endpoint of the controller, is that endpoint always retrieving the result from the db; or is there a cache in the controller that could return the result. Trying to cover my bases on what we can guarantee with the existence of an activation result https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519096220100 ---- 2020-03-06 18:27:35 UTC - Brendan Doyle: yea the invoker kafka queues just build up. if i remember correctly though depending on how backed up they are they might go unhealthy if the healthcheck can't run in a timeframe https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519255220200?thread_ts=1583436410.201700&cid=C3TPCAQG1 ---- 2020-03-06 18:30:49 UTC - Rodric Rabbah: for a blocking request, the result is usually coming from the invoker through kafka (this is called the active ack) however it is possible the result comes from the db if the active ack is delayed but somehow the result has been recorded in the db first https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519449221500 ---- 2020-03-06 18:31:05 UTC - Rodric Rabbah: in the controller there is a slow poll of the db while it’s waiting for the active ack to arrive https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519465221900 ---- 2020-03-06 18:31:48 UTC - Brendan Doyle: we're only doing non-blocking requests so is it always the db in that case? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519508222500 ---- 2020-03-06 18:32:17 UTC - Rodric Rabbah: for non blocking then yes, you’re retrieving results from the activation store (eg couch) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519537223100?thread_ts=1583519537.223100&cid=C3TPCAQG1 ---- 2020-03-06 18:32:30 UTC - Rodric Rabbah: and these are not cached iirc https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519550223400 ---- 2020-03-06 18:33:04 UTC - Brendan Doyle: great thanks https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519584223500?thread_ts=1583519537.223100&cid=C3TPCAQG1 ---- 2020-03-06 18:41:34 UTC - Dave Grove: i haven’t had a chance to look at elasticsearch PR, so just a couple of general wild guesses. First, be sure you’ve locally updated the imageTags for controller/invoker in values.yaml to pick up an image that was built after the PR was merged. Otherwise setting the options won’t do anything. Second, maybe try doing a kubectl exec into the invoker and controller pod and made sure you can curl to your ES that you are running in docker. By default the chart creates `NetworkPolicy` that might be getting in the way https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583520094223700?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 19:37:01 UTC - Bilal: Thanks! I did change the image tags to the most recent off the openwhisk dockerhub at the time `f061112` to be specific. <https://hub.docker.com/layers/openwhisk/invoker/f061112/images/sha256-05ccb957cadce3583af89109ca72ec4c3c1d8d2e4dba661744c1678cfb6a1192?context=explore> I didn't think about trying to poke ES manually from within the container and that worked out fine. I also POSTed a message into ES and was able to retrieve it. I am testing things out on KIND before I deploy to my on-prem kube cluster. Although I doubt that makes a difference. I will poke at this more early next week. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583523421223900?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 20:21:08 UTC - Bilal: ```kubectl -n openwhisk logs owdev-controller-0 Error: Could not find or load main class CONFIG_whisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider Caused by: java.lang.ClassNotFoundException: CONFIG_whisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526068224100?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 20:21:18 UTC - Bilal: So that's an issue When I replace `CONFIG_WHISK` with `-Dwhisk` it doesn't complain and I can see the string under invoker/controller opts, but it doesn't work. So I'm not sure if where I'm setting things is correct https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526078224300?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 20:24:17 UTC - Bilal: I don't think this is the proper way to set things https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526257224600?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 20:25:06 UTC - Bilal: Although when I `kubectl describe` I do see the values `CONTROLLER_OPTS: -Dwhisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider -Dwhisk_activationStore_elasticsearch_protocol=http -Dwhisk_activationStore_elasticsearch_hosts=172.18.0.1:9200 -Dwhisk_activationStore_elasticsearch_indexPattern=openwhisk -Dwhisk_activationStore_elasticsearch_username=admin -Dwhisk_activationStore_elasticsearch_password=admin` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526306225000?thread_ts=1583435988.200500&cid=C3TPCAQG1 ---- 2020-03-06 23:15:35 UTC - p-epp: Hello, I wanted to setup openwhisk with the kamon/metrics feature following this Doc: <https://github.com/apache/openwhisk/blob/master/docs/metrics.md>. Now, after enabling the configuration under ansible/environments/local/group_vars/all as you see in the picture, I get an Exception like this as it seems within the controller on loop: Exception in thread "Host Metrics" java.lang.NoClassDefFoundError: oshi.jna.platform.linux.Udev (initialization failure) at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:143) at oshi.hardware.platform.linux.LinuxDisks.getDisks(LinuxDisks.java:109) at oshi.hardware.platform.linux.LinuxHardwareAbstractionLayer.getDiskStores(LinuxHardwareAbstractionLayer.java:78) at kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask.recordStorageActivity(HostMetricsCollector.scala:177) at kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask.$anonfun$schedule$2(HostMetricsCollector.scala:131) at kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask$$Lambda$959/00000000DC00EEA0.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.concurrent.Future$$$Lambda$366/00000000B414AAC0.apply(Unknown Source) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.Future$$Lambda$367/00000000B414B060.apply(Unknown Source) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.Promise$$Lambda$283/000000002147ECC0.apply(Unknown Source) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'udev': libudev.so: cannot open shared object file: No such file or directory libudev.so: cannot open shared object file: No such file or directory Native library (linux-x86-64/libudev.so) not found in resource path [...] Am I missing something? Or did I break it? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583536535227000 ----