2020-03-06 01:46:56 UTC - Dominic Kim: It is briefly mentioned here: 
<https://github.com/apache/openwhisk-deploy-kube/blob/caac4d6de2193301c948b4091345224fc69fdbce/docs/configurationChoices.md#using-an-external-database>
 but I am not aware of any document.
I would defer to IBM guys who are operating OW on K8S.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583459216204900?thread_ts=1583279584.123800&cid=C3TPCAQG1
----
2020-03-06 02:32:32 UTC - Jiang PengCheng: the 
`CONFIG_whisk_activationStore_elasticsearch_hosts`  should not contain the 
`http://` or `https://` , just use ip address here, e.g. 
`192.168.1.1:9200,192.168.1.2:9200`
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583461952205100?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 02:33:01 UTC - Jiang PengCheng: and if your ES doesn't need any 
auth, you can provide any username/password
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583461981205300?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 02:34:28 UTC - Jiang PengCheng: I'm not familar with 
`openwhisk-deloy-kube repo` , it would be great if you can work it out and make 
a PR
+1 : Bilal
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583462068205500?thread_ts=1583430426.197100&cid=C3TPCAQG1
----
2020-03-06 04:58:33 UTC - chetanm: @Brendan Doyle You mean passing config via 
`-c` option?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583470713205700?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:02:36 UTC - Brendan Doyle: In the jar there’s just normal command 
line options, -m for example for the manifest. How do I set that when running 
the docker container, I don’t see any documentation on it because there’s just 
the run script for docker which doesn’t pass in any configs
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583470956208200?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:05:25 UTC - chetanm: With Docker passing external files would be 
tricky. You would need to mount the local directory as volume and then refer to 
mounted path
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471125208400?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:05:46 UTC - chetanm: The start scripts does pass the script args 
to Docker command 
<https://github.com/apache/openwhisk/blob/master/core/standalone/start.sh>
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471146208600?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:06:12 UTC - chetanm: But then you would need to craft the docker 
command manually to pass the volume settings
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471172208900?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:06:35 UTC - chetanm: For such cases I believe using the jar mode 
is better and convinient
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471195209100?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:06:43 UTC - chetanm: @Michele Sciabarra Thoughts?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583471203209300?thread_ts=1583439011.202600&cid=C3TPCAQG1
----
2020-03-06 05:38:50 UTC - Carlos Santana: out of `user-memory` is not 
indication that invokers are unhealthy just bussy, the activations would just 
seat on kafka topic waiting, from what I understand
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583473130210300?thread_ts=1583436410.201700&cid=C3TPCAQG1
----
2020-03-06 11:16:00 UTC - Tom Barber: I left OW alone for 24 hours and now it 
does this:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493360210700
----
2020-03-06 11:16:02 UTC - Tom Barber: ```MT-109900:~ barber$ wsk action delete 
/guest/pixlize-serverless-dev-triggerPiquantJob
error: Unable to delete action 'pixlize-serverless-dev-triggerPiquantJob': 
Unexpected http response code: 500 Internal Server Error (details: 
{"error":"unknown_error","reason":"undefined"}
) (code VXdgUj5Of9881NKLgHQy3Tvb4neBxZEd)
Run 'wsk --help' for usage.
MT-109900:~ barber$ wsk action delete /guest/pixlize-serverless-dev-testParams
error: Unable to delete action 'pixlize-serverless-dev-testParams': Unexpected 
http response code: 500 Internal Server Error (details: 
{"error":"unknown_error","reason":"undefined"}
) (code kQ0bBAPLTI4nCTIndcQdM2wVr2jq1e29)
Run 'wsk --help' for usage.```
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493362211000
----
2020-03-06 11:16:14 UTC - Tom Barber: any guesses
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493374211200
----
2020-03-06 11:18:10 UTC - Tom Barber: oh a crap load of Couchdb errors
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493490211500
----
2020-03-06 11:18:17 UTC - Tom Barber: :exploding_head:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493497211800
----
2020-03-06 11:18:51 UTC - Tom Barber: always couchdb that fails in some way or 
other
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493531212100
----
2020-03-06 11:21:21 UTC - Tom Barber: and then I restart couchdb and its 
completely hosed the install
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493681212500
----
2020-03-06 11:22:23 UTC - Tom Barber: thats even with a PVC in place though, I 
figured my couchdb woes would go away with some persistence
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493743213000
----
2020-03-06 11:23:26 UTC - Tom Barber: oh... the 2GB default volume size is full 
up
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493806213300
----
2020-03-06 11:23:29 UTC - Tom Barber: sad times
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583493809213500
----
2020-03-06 11:31:10 UTC - Tom Barber: alrighty
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583494270213800
----
2020-03-06 11:31:27 UTC - Tom Barber: hopelly 50Gi will last it a little longer 
than 2Gi :slightly_smiling_face:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583494287214300?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 11:45:23 UTC - Rodric Rabbah: Are you filling up with activations?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583495123214700?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:12:15 UTC - Bilal: I also had some issues with volume size on 
kube, I think it was for Kafka? Might have been the DB as well. I bumped up 
most of the internals to around that level, @Tom Barber, after talking to 
@Rodric Rabbah about it. There is a retention policy set per deployment that 
affects how long things are retained, but I haven't messed around with that as 
I don't need long term retention.

I had about 30,000 activations over the last 24 hours.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507535214900?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:17:17 UTC - Bilal: "Decoupling the database increases operational 
flexibility, for example by enabling blue/green deployments of OpenWhisk using 
a *shared database instance*." (emphasis added)
Aha, missed that bit when I read over this page a while ago. So I'd have two 
sets of openwhisk pointing at the same DB, then I can upgrade one OW deployment 
and if all is well make that live. Shouldn't be too hard, I already have a 
loadbalancer in front of everything pointing at my OW deployment, so it sounds 
like all I have to do is change where the loadbalancer points.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507837215200?thread_ts=1583279584.123800&cid=C3TPCAQG1
----
2020-03-06 15:17:50 UTC - Bilal: Well, I say "all I have to do" and make it 
sound simple, but there's always issues that pop up, lol
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507870215400?thread_ts=1583279584.123800&cid=C3TPCAQG1
----
2020-03-06 15:19:26 UTC - Bilal: Thanks! I also found out that `options` takes 
strings, and `CONFIG_WHISK` should be replaced with `-Dwhisk`
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507966215600?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 15:19:40 UTC - Bilal: Still messing around with it though
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583507980215800?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 15:21:30 UTC - Tom Barber: our stuff is minimal in usage currently 
just internal testing so nothing like @Bilal
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508090216100?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:22:11 UTC - Tom Barber: I blew it away and stuck 50Gb in, so I 
guess it wont occur again for now. Thanks for the pointers @Bilal I'll check 
the other volumes also to ensure I don't blow through them
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508131216300?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:25:04 UTC - Bilal: Well I'm still in internal testing phase too 
:slightly_smiling_face: I just had my OW listen to a bunch of stuff as a way 
prove stability while I continue to work on more stuff for my OW deployment. I 
think you are customizing your OW a lot more than me, but from what I've read 
sounds like you're working on a super cool project for NASA.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508304216500?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:33:35 UTC - Rodric Rabbah: <https://youtu.be/X2UaFuJsqxk>
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583508815216700?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 15:38:18 UTC - Bilal: Absolutely fantastic, thanks for sharing.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583509098217000?thread_ts=1583494287.214300&cid=C3TPCAQG1
----
2020-03-06 16:11:31 UTC - Bilal: ```      WHISK_LOGS_DIR:                       
                       
      CONFIG_whisk_info_date:                                      &lt;set to 
the key 'whisk_info_date' of config map 'owdev-whisk.config'&gt;  Optional: 
false
      CONFIG_whisk_couchdb_username:                               &lt;set to 
the key 'db_username' in secret 'owdev-db.auth'&gt;               Optional: 
false
      CONFIG_whisk_couchdb_password:                               &lt;set to 
the key 'db_password' in secret 'owdev-db.auth'&gt;               Optional: 
false
      CONFIG_whisk_couchdb_port:                                   &lt;set to 
the key 'db_port' of config map 'owdev-db.config'&gt;             Optional: 
false
      CONFIG_whisk_couchdb_protocol:                               &lt;set to 
the key 'db_protocol' of config map 'owdev-db.config'&gt;         Optional: 
false
      CONFIG_whisk_couchdb_host:                                   
owdev-couchdb.openwhisk.svc.cluster.local
      CONFIG_whisk_couchdb_provider:                               &lt;set to 
the key 'db_provider' of config map 'owdev-db.config'&gt;           Optional: 
false
      CONFIG_whisk_couchdb_databases_WhiskActivation:              &lt;set to 
the key 'db_whisk_activations' of config map 'owdev-db.config'&gt;  Optional: 
false
      CONFIG_whisk_couchdb_databases_WhiskEntity:                  &lt;set to 
the key 'db_whisk_actions' of config map 'owdev-db.config'&gt;      Optional: 
false
      CONFIG_whisk_couchdb_databases_WhiskAuth:                    &lt;set to 
the key 'db_whisk_auths' of config map 'owdev-db.config'&gt;        Optional: 
false```
More messing around/thinking outloud. So I think 
`CONFIG_whisk_couchdb_databases_WhiskActivation:`   needs to point to the 
elastic search whereas the `WhiskEntitiy` and `WhiskAuth` will remain in couchDB
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511091217200?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 16:12:14 UTC - Bilal: and I do see my invoker options being set
```INVOKER_OPTS:                                                
-Dwhisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider
 -Dwhisk_activationStore_elasticsearch_protocol=http 
-Dwhisk_activationStore_elasticsearch_hosts=172.18.0.1:9200 
-Dwhisk_activationStore_elasticsearch_indexPattern=openwhisk 
-Dwhisk_activationStore_elasticsearch_username=admin 
-Dwhisk_activationStore_elasticsearch_password=admin  
-Dkubernetes.master=https://$KUBERNETES_SERVICE_HOST 
-Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider```
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511134217400?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 16:24:40 UTC - Bilal: Oh oh oh, the `openwhisk-deploy-kube` repo 
pulls ansible playbooks from the main `openwhisk` repo when the CouchDB is 
initialized
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583511880217600?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 17:46:34 UTC - Bilal: @Dave Grove Any advice or direction on how to 
use <https://github.com/apache/openwhisk/pull/4724> with the 
openwhisk-deploy-kube/helm?. Unfortunately, I've never used ansible before. I 
feel like I understand what needs to happen but I can't wrap my head around 
how. I've poked around in the changes made in the PR and from what I 
understand, the CouchDB gets init and created using the same ansible playbook 
in both the helm/kube deployment and the ansible deployment. Where it diverges 
is the nextstep in the readme from the ElasticSearch PR  `ansible-playbook -i 
environments/&lt;environment&gt; openwhisk.yml -e 
db_activation_backend=ElasticSearch` The backend gets set to ElasticSearch in 
`ansible/group_vars/all` and the only thing I can see that interacts with this 
setting are the `deploy.yml` playbooks for both the invoker and controller 
which sets the options above that Jiang PengCheng helped me out with.

I'm a bit stuck on what to try, any help is appreciated, thanks!
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583516794217800?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 18:24:56 UTC - Brendan Doyle: quick question. when we do a get on an 
activation for the result through the http endpoint of the controller, is that 
endpoint always retrieving the result from the db; or is there a cache in the 
controller that could return the result. Trying to cover my bases on what we 
can guarantee with the existence of an activation result
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519096220100
----
2020-03-06 18:27:35 UTC - Brendan Doyle: yea the invoker kafka queues just 
build up. if i remember correctly though depending on how backed up they are 
they might go unhealthy if the healthcheck can't run in a timeframe
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519255220200?thread_ts=1583436410.201700&cid=C3TPCAQG1
----
2020-03-06 18:30:49 UTC - Rodric Rabbah: for a blocking request, the result is 
usually coming from the invoker through kafka (this is called the active ack)
however it is possible the result comes from the db if the active ack is 
delayed but somehow the result has been recorded in the db first
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519449221500
----
2020-03-06 18:31:05 UTC - Rodric Rabbah: in the controller there is a slow poll 
of the db while it’s waiting for the active ack to arrive
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519465221900
----
2020-03-06 18:31:48 UTC - Brendan Doyle: we're only doing non-blocking requests 
so is it always the db in that case?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519508222500
----
2020-03-06 18:32:17 UTC - Rodric Rabbah: for non blocking then yes, you’re 
retrieving results from the activation store (eg couch)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519537223100?thread_ts=1583519537.223100&cid=C3TPCAQG1
----
2020-03-06 18:32:30 UTC - Rodric Rabbah: and these are not cached iirc
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519550223400
----
2020-03-06 18:33:04 UTC - Brendan Doyle: great thanks
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583519584223500?thread_ts=1583519537.223100&cid=C3TPCAQG1
----
2020-03-06 18:41:34 UTC - Dave Grove: i haven’t had a chance to look at 
elasticsearch PR, so just a couple of general wild guesses.
First, be sure you’ve locally updated the imageTags for controller/invoker in 
values.yaml to pick up an image that was built after the PR was merged.  
Otherwise setting the options won’t do anything.
Second, maybe try doing a kubectl exec into the invoker and controller pod and 
made sure you can curl to your ES that you are running in docker.  By default 
the chart creates `NetworkPolicy`  that might be getting in the way
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583520094223700?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 19:37:01 UTC - Bilal: Thanks! I did change the image tags to the 
most recent off the openwhisk dockerhub at the time `f061112` to be specific. 
<https://hub.docker.com/layers/openwhisk/invoker/f061112/images/sha256-05ccb957cadce3583af89109ca72ec4c3c1d8d2e4dba661744c1678cfb6a1192?context=explore>

I didn't think about trying to poke ES manually from within the container and 
that worked out fine. I also POSTed a message into ES and was able to retrieve 
it.
I am testing things out on KIND before I deploy to my on-prem kube cluster.  
Although I doubt that makes a difference.

I will poke at this more early next week.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583523421223900?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 20:21:08 UTC - Bilal: ```kubectl -n openwhisk logs owdev-controller-0
Error: Could not find or load main class 
CONFIG_whisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider
Caused by: java.lang.ClassNotFoundException: 
CONFIG_whisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider```
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526068224100?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 20:21:18 UTC - Bilal: So that's an issue
When I replace `CONFIG_WHISK` with `-Dwhisk` it doesn't complain and I can see 
the string under invoker/controller opts, but it doesn't work. So I'm not sure 
if where I'm setting things is correct
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526078224300?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 20:24:17 UTC - Bilal: I don't think this is the proper way to set 
things
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526257224600?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 20:25:06 UTC - Bilal: Although when I `kubectl describe` I do see 
the values
`CONTROLLER_OPTS:                                             
-Dwhisk_spi_ActivationStoreProvider=org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider
 -Dwhisk_activationStore_elasticsearch_protocol=http 
-Dwhisk_activationStore_elasticsearch_hosts=172.18.0.1:9200 
-Dwhisk_activationStore_elasticsearch_indexPattern=openwhisk 
-Dwhisk_activationStore_elasticsearch_username=admin 
-Dwhisk_activationStore_elasticsearch_password=admin`
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583526306225000?thread_ts=1583435988.200500&cid=C3TPCAQG1
----
2020-03-06 23:15:35 UTC - p-epp: Hello, I wanted to setup openwhisk with the 
kamon/metrics feature following this Doc: 
<https://github.com/apache/openwhisk/blob/master/docs/metrics.md>. Now, after 
enabling the configuration under ansible/environments/local/group_vars/all as 
you see in the picture, I get an Exception like this as it seems within the 
controller on loop:

Exception in thread "Host Metrics" java.lang.NoClassDefFoundError: 
oshi.jna.platform.linux.Udev (initialization failure)
        at 
java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:143)
        at oshi.hardware.platform.linux.LinuxDisks.getDisks(LinuxDisks.java:109)
        at 
oshi.hardware.platform.linux.LinuxHardwareAbstractionLayer.getDiskStores(LinuxHardwareAbstractionLayer.java:78)
        at 
kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask.recordStorageActivity(HostMetricsCollector.scala:177)
        at 
kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask.$anonfun$schedule$2(HostMetricsCollector.scala:131)
        at 
kamon.instrumentation.system.host.HostMetricsCollector$InfrequentCollectionTask$$Lambda$959/00000000DC00EEA0.apply$mcV$sp(Unknown
 Source)
        at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
        at scala.concurrent.Future$$$Lambda$366/00000000B414AAC0.apply(Unknown 
Source)
        at scala.util.Success.$anonfun$map$1(Try.scala:255)
        at scala.util.Success.map(Try.scala:213)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
        at scala.concurrent.Future$$Lambda$367/00000000B414B060.apply(Unknown 
Source)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
        at 
scala.concurrent.impl.Promise$$Lambda$283/000000002147ECC0.apply(Unknown Source)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'udev':
libudev.so: cannot open shared object file: No such file or directory
libudev.so: cannot open shared object file: No such file or directory
Native library (linux-x86-64/libudev.so) not found in resource path [...]

Am I missing something? Or did I break it?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1583536535227000
----

Reply via email to