2019-02-28 01:36:18 UTC - Dominic Kim: Hello guys
wave : Carlos Santana, James Thomas
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551317778180100
----
2019-02-28 02:24:02 UTC - Perry Dykes: Folks, I need to process an incoming xml
stream. The legacy clients that uploads a Post XML file can't be changed, and
attempting OpenWhisk to process using the raw web action function. I have
created my tests using in the docs at
<https://console.bluemix.net/docs/openwhisk/openwhisk_webactions.html#openwhisk_webactions>.
I did have to change the Javascript (using Node) due new Buffer() deprecation:
```
function main(args) {
var decoded = Buffer.from(args.__ow_body, 'base64');
decoded = decoded.toString('utf-8');
console.log("incoming text", decoded);
return { "body": decoded };
}
```
I set the endpoint options to Enable as Web Action and Raw HTTP Handling. Then
used the curl invocation (and several variants ;-)) to get a feel for this the
raw handling. In net, trying to upload an XML file and processes, but stuck at
this step processing a simple string, let alone an xml snippet.
curl -k -H "content-type: application" -X POST -d "some test string to dump
out" <url to action>
However, this documentation snippet example seems to simply fail with much
error that makes sense, e.g. "The request's Content-Type is not supported.
Expected:\napplication/json"}". I had thought the reason for raw was to be
able to take a text string. Anyway, missing some newbie thing I guess, tips or
links to a xml processing sample and/or ideas on the above appreciated.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551320642185600
----
2019-02-28 02:26:46 UTC - Rodric Rabbah: What url are you using
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551320806186200
----
2019-02-28 03:23:27 UTC - Perry Dykes: @Rodric Rabbah -u <api-key>
<https://eu-de.functions.cloud.ibm.com/api/v1/namespaces/IBMSystemsSupport_dev/actions/tss_mvs/CiscoAlert>
(basically the url provided in the Endpoints CURL example)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551324207187300
----
2019-02-28 03:38:11 UTC - Rodric Rabbah: A web action has a different url - if
the action is a web action, `wsk action get <name> --url` will show you
the url to use
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325091188800
----
2019-02-28 03:43:34 UTC - Rodric Rabbah: It has a “web/” in the path.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325414189300
----
2019-02-28 03:45:37 UTC - Perry Dykes: Hmm, ok, anoher public method
unprotected is here:
<https://eu-de.functions.cloud.ibm.com/api/v1/web/IBMSystemsSupport_dev/tss_mvs/CiscoAlert.json>
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325537189900
----
2019-02-28 03:46:57 UTC - Perry Dykes: Ok, that does work vs. the url I was
trying.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325617190400
----
2019-02-28 03:47:27 UTC - Perry Dykes: Guess I have to read up on the
differences.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325647190800
----
2019-02-28 03:47:48 UTC - Perry Dykes: seems odd to require I end a uri in
.json doesn't it?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325668191200?thread_ts=1551325668.191200&cid=C3TPCAQG1
----
2019-02-28 03:49:50 UTC - Perry Dykes: Thanks for the pointer, that does allow
me to process the xml as I had hoped.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551325790191600
----
2019-02-28 05:50:58 UTC - Dominic Kim: Is anyone facing this issue?
-1 : Rodric Rabbah
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551333058000200
----
2019-02-28 08:03:41 UTC - chetanm: Currently if we reset any namespace key the
old keys are still valid for some time due to cache effect.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341021001100
----
2019-02-28 08:04:29 UTC - chetanm: Should the cache entry for Identity be also
have invalidation support such that upon expiry/invalidation of any namespace
key system admin service can trigger a cache invalidation message?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341069002100
----
2019-02-28 08:07:17 UTC - Markus Thömmes: @chetanm it could, however the
controller does not know about that invalidation, does it?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341237002700
----
2019-02-28 08:08:14 UTC - chetanm: yup … so whatever admin service is being
used by system which does the namespace invalidation would need to push an
explicit message to `cacheInvalidation` topic
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341294003600
----
2019-02-28 08:08:44 UTC - chetanm: This would though not work where we are
directly using wskadmin python script
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341324004000
----
2019-02-28 08:08:52 UTC - chetanm: may be for wskadmin-next we can add
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341332004300
----
2019-02-28 08:09:25 UTC - Markus Thömmes: ooooor use this as a segway for your
"cache invalidation through database changes feed" proposal?
:slightly_smiling_face:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341365004700
----
2019-02-28 08:10:15 UTC - chetanm: That was a by product I was looking for
:slightly_smiling_face:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341415005300
----
2019-02-28 08:10:47 UTC - chetanm: In testing we observed that invalidation is
not effective immediately hence was thinking that would also get solved once we
have such invalidation support
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341447006000
----
2019-02-28 08:11:06 UTC - chetanm: but realized that current logic only
invalidates `whisks` related caches
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341466006500
----
2019-02-28 08:11:43 UTC - Markus Thömmes: I'd rather invest in that vs.
investing in adding kafka stuff to wskadmin
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341503007200
----
2019-02-28 08:12:32 UTC - chetanm: sure … so add support in
`RemoteCacheInvalidation` to look for message to invalidate entries from
Identity cache also
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341552008000
----
2019-02-28 08:12:46 UTC - chetanm: and then have a feed support to push them
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341566008500
----
2019-02-28 08:12:50 UTC - Markus Thömmes: now, invalidation will never be
immediately consistent unless you wait for all controllers to have picked it up
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341570008600
----
2019-02-28 08:13:42 UTC - chetanm: yeah … just that with namespace and security
… we may like to make it earlier …. anyways it would expire after 5 mins. So
current situation is somewhat ok
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341622009600
----
2019-02-28 08:13:57 UTC - Carlos Santana: Don’t use an extension it would be
the same a .http
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341637010200?thread_ts=1551325668.191200&cid=C3TPCAQG1
----
2019-02-28 08:15:01 UTC - Markus Thömmes: makes sense
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551341701010900
----
2019-02-28 09:20:58 UTC - Rodric Rabbah: Is setting up a feed more work vs just
posting an explicit message to Kafka - I mean that we already have code for
doing that which wskadmin-next can use don’t we?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551345658013700
----
2019-02-28 09:29:35 UTC - Markus Thömmes: It's more of a generalization concern
on my end. The feed work is valuable in a more general way. With Kafka
messages, you need to make sure the admin tool actually runs in-cluster etc etc
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346175014400
----
2019-02-28 09:31:42 UTC - chetanm: feed is lot more generic. Further I am not
sure how system operators are handling the user part (I presume wskadmin is
more for dev purpose). For our case we have a separate service which
manipulates the user db
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346302015600
----
2019-02-28 09:33:56 UTC - Markus Thömmes: so a changes feed answers the more
general question of "inform me whenever something happens in the database",
which can be useful for all kinds of database manipulations (although that
should be discouraged of course)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346436016500
----
2019-02-28 09:43:18 UTC - Rodric Rabbah: what about just an http request - I’m
just thinking - if this is a low probability event - is a much simpler solution
is adequate vs a more general but potentially laborious solution
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551346998018700
----
2019-02-28 09:45:06 UTC - Rodric Rabbah: (i will read the proposal in the wiki
- I haven’t yet)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347106021700
----
2019-02-28 09:45:55 UTC - chetanm: @Rodric Rabbah Feed approach here is in
context of a broader usecase around multi region support.
<https://github.com/apache/incubator-openwhisk/issues/4277#cache-invalidation-service>
While doing that I thought to use same approach to invalidate namespace keys
upon invalidation … whihc started this thread
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347155022800
----
2019-02-28 09:48:47 UTC - Rodric Rabbah: :+1: clearly I should have read the
wiki first
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551347327023400
----
2019-02-28 10:04:41 UTC - Dominic Kim: Is it possible for other databases which
do not support something like `_changes` API in CouchDB to relay on the "feed"?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551348281025400
----
2019-02-28 10:05:29 UTC - Dominic Kim: I hope this is handled in a
database-independent way.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551348329026100
----
2019-02-28 10:27:36 UTC - Markus Thömmes: @Dominic Kim which database would
come to your mind? Relational databases usually have a "trigger" concept for
that kind of thing
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349656027000
----
2019-02-28 10:28:25 UTC - Dominic Kim: Actually, we here are using
ElasticSearch as a database.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349705027500
----
2019-02-28 10:30:03 UTC - Markus Thömmes: I'd guess we'd have an interface to
implement per database to obtain that kind of information. We could leave Kafka
based invalidation in-place as well
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349803028200
----
2019-02-28 10:32:38 UTC - Dominic Kim: That means, in case the database does
not support any event notification feature, it can still rely on Kafka based
invalidation?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349958030800
----
2019-02-28 10:33:06 UTC - chetanm: To clarify some aspect - Proposal here was
more for invalidating cache in multi cluster setup which share same DB. So idea
here was (in case of CosmosDB) we can make use of change feed to detect change
in db and convert them as events to each clusters `cacheInvalidation` topic. So
kafka based invalidation is still used
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551349986031300
----
2019-02-28 10:33:26 UTC - Dominic Kim: Yes.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350006031800
----
2019-02-28 10:34:12 UTC - chetanm: As those multi clusters do not share kafka
we need a way to ensure that changes made in entities are flushed out. Which
work fine in single cluster setup as Controller which handled the change pushes
the event to `cacheInvalidation` topic.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350052033200
----
2019-02-28 10:34:42 UTC - chetanm: In MR setups this needs to be done by a
separate service which uses some sort of change feed if possible. So solution
would be db specific
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350082033900
----
2019-02-28 10:36:28 UTC - Dominic Kim: @chetanm So you keep the independent
service which listens on the changes event from the database in your mind?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350188034800
----
2019-02-28 10:36:42 UTC - chetanm: yup
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350202035000
----
2019-02-28 10:36:46 UTC - Dominic Kim: got it.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350206035200
----
2019-02-28 10:36:59 UTC - Dominic Kim: I thought controllers are receiving that
kinds of events.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350219035600
----
2019-02-28 10:37:53 UTC - Dominic Kim: In that case, we can just implement the
separate service to achieve the same thing though the database does not support
such "feed" feature. Maybe relying on long-polling or something.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350273037300
----
2019-02-28 10:38:12 UTC - chetanm: So controller which say recived the change
request for action A1 would make change to db and would also push an event to
cacheInvalidation topic. This would then be picked by other Controllers and
they would evict that entry from there respective caches
white_check_mark : Dominic Kim
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350292037700
----
2019-02-28 10:39:03 UTC - chetanm: Now when we have multiple such clusters then
we need a way to project such events in other cluster (separate from one where
the actual change happened)
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350343038600
----
2019-02-28 10:39:30 UTC - chetanm: Thats where this service would come in and
do this db change feed to cache invalidation event conversion
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350370039200
----
2019-02-28 10:39:55 UTC - Dominic Kim: Yes because each cluster does not share
the same Kafka.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350395039800
----
2019-02-28 10:40:30 UTC - Markus Thömmes: For full disclosure: My proposal
towards that was to ditch the Kafka topic entirely and only rely on database
based notification for cache invalidation period
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350430041000
----
2019-02-28 10:40:33 UTC - chetanm: Yes. Other approach can be to replicate the
kafka cluster topic … which in our case we are avoiding as Cosmosdb serve
similar purpose easily
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350433041200
----
2019-02-28 10:41:09 UTC - Markus Thömmes: I think it's much more elegant than
the current Kafka design
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350469041500
----
2019-02-28 10:41:55 UTC - Dominic Kim: @Markus Thömmes Then you meant, we can
relay on Kafka iff the database does not support such a feature?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350515042200
----
2019-02-28 10:42:13 UTC - Markus Thömmes: if we make it pluggable in the right
spots, sure
white_check_mark : Dominic Kim
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350533042600
----
2019-02-28 10:42:38 UTC - Markus Thömmes: I'd prefer doing that over creating a
seperate service that transforms db events to kafka events
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350558043100
----
2019-02-28 10:44:12 UTC - chetanm: hmm … if each controller pulls in the change
feed then it would put load on db which incurs cost in ComsosDB. So using kafka
for local cache sync and feed for cross cluster may work better
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350652044300
----
2019-02-28 10:48:33 UTC - Markus Thömmes: The changes feed is expensive?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551350913044500
----
2019-02-28 11:41:17 UTC - chetanm: It has some cost. Each document read adds to
the read units consumed. Further each feed consumer needs to manage it's offset
... Kind of similar to how Kafka consumer works. The offset are again stored in
a separate collection in CosmosDB
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551354077046600
----
2019-02-28 15:00:59 UTC - Olivier Tardieu: openwhisk-composer 0.10.0 has been
released and is available from npm and github:
Changelog:
- Add new parallel and map combinators to run compositions in parallel using a
Redis instance to store intermediate results.
- Add dynamic combinator to invoke an action with a name chosen at run time.
- Add option to bypass TLS certificate validation failures (off by default).
- Add API to generate the conductor action code from a composition.
- Add control over needle options and logging.
Download links:
- <https://www.npmjs.com/package/openwhisk-composer>
- <https://github.com/apache/incubator-openwhisk-composer/releases>
- <http://openwhisk.incubator.apache.org/downloads.html>
tada : Carlos Santana, Matt Sicker, James Thomas, chetanm, mmarth, seonghyun,
dubee
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1551366059047800
----
