Re: Re: Backpressure for slow activation storage in Invoker
Let me share a few ideas on them. Regarding option B1, I think it can scale out better than option B2. If I understood correctly, scaling out of the service will be highly dependent on Kafka. Since the number of consumers is limited to the number of partitions, the number of service nodes will be also limited to the number of partitions. So in the case of B2, if we create a new topic with some partition numbers, we cannot scale out the service nodes more than that. At some point, we may need to alter the number of partitions and it's not easy in Kafka. (Since the activation processing here is asynchronous, we may bear some downtime(1~2s) to alter the partition. Then it would be fine.) In the case of B1, there will be many controller topics with their own partitions. Since controllers can be scaled out, there will be more topics, and the activation service can scale out accordingly. But in this case, we need to manually control the topic assignment. (Not partition assignment, it will be done by Kafka.) Let's say we have 3 controller topics with 2 partitions each. For HA, it would be great to have at least two nodes. At first, both nodes will take care of all three topics. Based on the partition assignment plan in Kafka, both nodes will fetch activation messages without any duplication. As controllers are scaled out, two nodes may not be enough to take care of all topics. At this point, we need to scale out the service nodes more. Then we need to do logical partitioning for topics. For example, the node1 and 2 will take care of topic0 ~ 1 and node3 and 4 will take care of topic2 ~ 3. In this way, we can guarantee the minimum HA and scale out the nodes as well. Among them, topic partitions will be also assigned by Kafka. So in short, For B1, we can scale out the service as controllers are scaled out, but it would be much complex to manually assign topics. And one node may have more than one Kafka consumers. For B2, scaling might be limited unless we have a big enough number of partitions at topic creation time. But if we can bear some downtime, this might not be a problem and this option will be a lot simpler. Best regards Dominic. 2019년 6월 24일 (월) 오후 6:50, Chetan Mehrotra 님이 작성: > Okie so we can then aim for adding an optional support for storing > activations via a separate service. > > Currently we also send the activation result on respective controller > topic. With this change we would also be sending same activation > record on another topic. So we have another choice to make > > Option B1 - Activations via controller topic > > > Here we avoid a new topic and instead have a service which listen to > all controller topics for activation records. However that would be > tricky to implement and also tricky to scale out. As scaling out such > a service by running multiple copies would not be easy in terms of > sharding/partitioning > > Here the benefit is that we reduce the duplicate writes on Kafka. > > Option B2 - Introduce a new topic altogether > --- > > We introduce a new topic to which all invokers write the activation > records (like the case for user-events). Then implementing a new > service to read from a single (possibly partitioned topic) would be > easier. > > My suggestion is to go for B2 for now. > > Any feedback on that? > > Chetan Mehrotra > > On Fri, Jun 21, 2019 at 11:46 PM Rodric Rabbah wrote: > > > > > Can we handle these in same way as user events? Maybe exactly like user > > events, as in use a single service to process both topics. > > > > good call - the user events already contains much of the activation > record > > (if not all modulo the logs)? > > > > -r >
Re: oauth token verification in the controller
It might not be directly related to the agenda but one small thing to note is the response time of the auth/authz server should be considered as well. When I implemented LDAP integration, the LDAP server was too slow. It took about 100ms ~ 1s to authenticate users. Even though I took advantage of the cache, I could not reduce the cold start(in terms of authentication) latency, and I had to drop the LDAP integration in the end. In my case, LDAP is legacy, I didn't have many choices. So if the authentication backend is not fast as enough, you might need to ponder some ways to overcome it. Regards Dominic 2019년 6월 24일 (월) 오전 1:56, Martin Henke 님이 작성: > Rodric, > > IBM Cloud Functions hasn’t implemented OAuth, but JWT based bearer token > based authentication based on the IBM Cloud IAM system. > > The first problem we encountered was that our bearer token did not provide > a namespace related context > (which is only configured in the IAM system). The only obvious namespace > related information is contained in the URI (but unfortunately not always > when looking at the _ default). > > The next problem was the need to construct the Identity object (of which > the namespace information is a key component) fully in the authorization > code to not break the API handling (which brings us back to the missing > namespace information). > > We overcame the problems by asking the user to provide missing namespace > information via additional headers and by retrieving additional > authorization related data from proprietiary data stores. > > There were more problems in the realm to provide the meaningful error > messages. > Nevertheless our implementation proves that OW is already able to handle > JWT based bearer tokens and provide them to be used in outbound calls. > > Regards, > Martin > > > Am 21.06.2019 um 13:23 schrieb Rodric Rabbah : > > > > I'm curious if anyone has thought about or implemented an oauth based > > authentication mechanism in the controller. I've thought about replacing > > the subject authentication with oauth and think it would not be a lot of > > work to do although it does have some wider implications. > > > > -r > >
exporting activation arguments to the environment
In the current activation model, an action's arguments are always provided
to the action on "run", not "init".
Should we consider partitioning the argument list into two sets, the first
is exported as environment variables at "init" time, and the second become
the action's argument at "run" time? A criteria for partitioning is that
the environment variable starts with a capital letter, which is a common
convention.
For example, an action which is invoked with a JSON object
{ "XYZ": true,
"abc" : false }
would receive {"abc": false} as its arguments and can read XYZ from the
environment (as process.env.XYZ == "true" in Node.js).
This change would:
1. require a change in the invoker to pass arguments during initialization
2. require a change in the runtime proxies to export the arguments to the
environment at initialization time (additional work may be implied by 1b)
3. an annotation on actions to opt into this partitioning for backward
compatibility or to opt out. For example '-a env-partition-arguments true'
partitions the arguments and actions without this annotation are not
affected.
Some obvious question:
Q1a. should the invoker perform the partitioning or delegate it to the
runtime? The advantage of the former is that the runtimes do not have to
implement the filtering policy and do less work. I think it makes sense to
do this invoker side for uniformity.
Q1b. should the partitioning treat environment variables as immutable post
init and ignore the partition on warm starts? This is an issue when a value
is overridden during POST invoke only since for a webaction, you cannot
override a value that's already defined (and updating a bound parameter on
an action invalidates warm containers). I think env vars should be treated
as immutable despite the issue with POST invoke.
-r
[GitHub] [incubator-openwhisk-pluggable-provider] jthomas merged pull request #8: Fixing scancode issues.
jthomas merged pull request #8: Fixing scancode issues. URL: https://github.com/apache/incubator-openwhisk-pluggable-provider/pull/8 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Openwhisk in a standalone runnable jar (#4516)
oh, one thing worth mentioning: when testing on electron, on linux, i have to use an http keep-alive http agent when talking to the standalone. this has been a perennial problem with electron/linux versus openwhisk (i still have no idea why); but the standalone jar made the problem considerably worse. i don't know whether this is an openwhisk bug or a linux bug or an electron bug, but something breaks if electron+linux makes too many connections to openwhisk. On Mon, Jun 24, 2019 at 8:31 AM Nick Mitchell wrote: > it's been working well for our CI/CD tests. standalone openwhisk, combined > with microk8s, are a perfect combination for test pipelines against this > kind of infrastructure. > > thanks!! > > On Mon, Jun 24, 2019 at 6:08 AM Chetan Mehrotra > wrote: > >> Hi Team, >> >> At this stage the OpenWhisk Standalone jar works fine. So would like >> to get it tried out in real world and for that need some volunteers! >> >> You can follow the steps at >> https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 >> and download the jar and try out a hello world or better some complex >> scenarios. >> >> In brief following steps would be needed to run this locally on your >> systems. >> >> $ wget >> https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar >> $ java -jar openwhisk-standalone.jar >> $ wsk property set --apihost 'http://localhost:3233' --auth >> >> '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP' >> >> Let me know your experience. If you face any issue or have some >> feedback then please share that on this thread or at [1]. Also check >> the Known Issues section at [2] >> >> Chetan Mehrotra >> [1] https://github.com/apache/incubator-openwhisk/pull/4516 >> [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 >> >> On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra >> wrote: >> > >> > Hi Nick, >> > >> > > we have the need to create namespaces/authkeys >> > >> > I have added an initial readme with steps on how to register custom >> > namespace at [1]. Can you give it a try and let us know if it meet >> > your requirements. Otherwise we can look into support a minimal REST >> > api to add user in standalone case >> > >> > Chetan Mehrotra >> > [1] >> https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces >> > >> > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell >> wrote: >> > > >> > > we have the need to create namespaces/authkeys (so that we can run >> tests in >> > > parallel); is this supported by the current build env? or would we >> have to >> > > launch a separate openwhisk process for each tenant? >> > > >> > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana >> wrote: >> > > >> > > > Chetan thanks for thinking on Windows users +1 >> > > > >> > > > Just curious why the jar can’t be build in Windows can you log an >> issue >> > > > for this? >> > > > >> > > > My 2 cents for Windows or Mac I think the best way to build the jar >> would >> > > > be inside a docker container. A one liner with a single command >> should be >> > > > able to produce the jar on a local workstation or devops pipeline. >> > > > >> > > > >> > > > - Carlos Santana >> > > > @csantanapr >> > > > >> > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra < >> chetan.mehro...@gmail.com> >> > > > wrote: >> > > > > >> > > > > Just to share some more progress. With runnable jar its now >> possible >> > > > > to run OpenWhisk standalone even on Windows and execute basic >> actions. >> > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10 >> > > > > >> > > > > Note you would probably not be able to build the project on >> windows. >> > > > > But copying the produced jar would let you run it on Windows. >> > > > > >> > > > > Chetan Mehrotra >> > > > > >> > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra >> > > > > wrote: >> > > > >> >> > > > >> Hi Nick, >> > > > >> >> > > > >> For now just building the PR branch and running following command >> > > > >> should get you going >> > > > >> >> > > > >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar >> > > > >> >> > > > >> You can use the default guest and whisk.system credentials to >> interact >> > > > >> with it. I am in the process of writing a readme for various >> options >> > > > >> exposed. Hope to get it done by Monday >> > > > >> >> > > > >> Chetan Mehrotra >> > > > >> >> > > > >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell < >> moose...@gmail.com> >> > > > wrote: >> > > > >>> >> > > > >>> thanks chetan for doing this! >> > > > >>> >> > > > >>> could you provide some example startup sequences, e.g. with >> sample >> > > > configs? >> > > > >>> i'm willing to try this out for our CI/CD pipeline -- i'm sick >> of 1) >> > > > >>> waiting 5-7 minutes for openwhisk to start up; and b) fighting >> ansible >> > > > >>> versus xenial :) >> > > > >>> >> > > > >>> @starpit >>
Re: Openwhisk in a standalone runnable jar (#4516)
it's been working well for our CI/CD tests. standalone openwhisk, combined with microk8s, are a perfect combination for test pipelines against this kind of infrastructure. thanks!! On Mon, Jun 24, 2019 at 6:08 AM Chetan Mehrotra wrote: > Hi Team, > > At this stage the OpenWhisk Standalone jar works fine. So would like > to get it tried out in real world and for that need some volunteers! > > You can follow the steps at > https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 > and download the jar and try out a hello world or better some complex > scenarios. > > In brief following steps would be needed to run this locally on your > systems. > > $ wget > https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar > $ java -jar openwhisk-standalone.jar > $ wsk property set --apihost 'http://localhost:3233' --auth > > '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP' > > Let me know your experience. If you face any issue or have some > feedback then please share that on this thread or at [1]. Also check > the Known Issues section at [2] > > Chetan Mehrotra > [1] https://github.com/apache/incubator-openwhisk/pull/4516 > [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 > > On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra > wrote: > > > > Hi Nick, > > > > > we have the need to create namespaces/authkeys > > > > I have added an initial readme with steps on how to register custom > > namespace at [1]. Can you give it a try and let us know if it meet > > your requirements. Otherwise we can look into support a minimal REST > > api to add user in standalone case > > > > Chetan Mehrotra > > [1] > https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces > > > > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell > wrote: > > > > > > we have the need to create namespaces/authkeys (so that we can run > tests in > > > parallel); is this supported by the current build env? or would we > have to > > > launch a separate openwhisk process for each tenant? > > > > > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana > wrote: > > > > > > > Chetan thanks for thinking on Windows users +1 > > > > > > > > Just curious why the jar can’t be build in Windows can you log an > issue > > > > for this? > > > > > > > > My 2 cents for Windows or Mac I think the best way to build the jar > would > > > > be inside a docker container. A one liner with a single command > should be > > > > able to produce the jar on a local workstation or devops pipeline. > > > > > > > > > > > > - Carlos Santana > > > > @csantanapr > > > > > > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra < > chetan.mehro...@gmail.com> > > > > wrote: > > > > > > > > > > Just to share some more progress. With runnable jar its now > possible > > > > > to run OpenWhisk standalone even on Windows and execute basic > actions. > > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10 > > > > > > > > > > Note you would probably not be able to build the project on > windows. > > > > > But copying the produced jar would let you run it on Windows. > > > > > > > > > > Chetan Mehrotra > > > > > > > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra > > > > > wrote: > > > > >> > > > > >> Hi Nick, > > > > >> > > > > >> For now just building the PR branch and running following command > > > > >> should get you going > > > > >> > > > > >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar > > > > >> > > > > >> You can use the default guest and whisk.system credentials to > interact > > > > >> with it. I am in the process of writing a readme for various > options > > > > >> exposed. Hope to get it done by Monday > > > > >> > > > > >> Chetan Mehrotra > > > > >> > > > > >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell < > moose...@gmail.com> > > > > wrote: > > > > >>> > > > > >>> thanks chetan for doing this! > > > > >>> > > > > >>> could you provide some example startup sequences, e.g. with > sample > > > > configs? > > > > >>> i'm willing to try this out for our CI/CD pipeline -- i'm sick > of 1) > > > > >>> waiting 5-7 minutes for openwhisk to start up; and b) fighting > ansible > > > > >>> versus xenial :) > > > > >>> > > > > >>> @starpit > > > > >>> > > > > >>> On Fri, Jun 21, 2019 at 12:44 AM Chetan Mehrotra < > > > > chetan.mehro...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > > What's the performance like on startup time > > > > > > > > It starts in < 5 secs. > > > > > > > > So far no major blocking issue apart from fetching docker logs > on Mac. > > > > Current approach of directly reading the log json does not > work. So > > > > need to have a mac version which uses `docker logs` command to > > > > somewhat handle such a case. > > > > > > > > Chetan Mehrotra > > > > > > > > > On Thu, Jun 20, 2019 at 9:50 PM James
Re: Openwhisk in a standalone runnable jar (#4516)
Hi Team, At this stage the OpenWhisk Standalone jar works fine. So would like to get it tried out in real world and for that need some volunteers! You can follow the steps at https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 and download the jar and try out a hello world or better some complex scenarios. In brief following steps would be needed to run this locally on your systems. $ wget https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar $ java -jar openwhisk-standalone.jar $ wsk property set --apihost 'http://localhost:3233' --auth '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP' Let me know your experience. If you face any issue or have some feedback then please share that on this thread or at [1]. Also check the Known Issues section at [2] Chetan Mehrotra [1] https://github.com/apache/incubator-openwhisk/pull/4516 [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10 On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra wrote: > > Hi Nick, > > > we have the need to create namespaces/authkeys > > I have added an initial readme with steps on how to register custom > namespace at [1]. Can you give it a try and let us know if it meet > your requirements. Otherwise we can look into support a minimal REST > api to add user in standalone case > > Chetan Mehrotra > [1] > https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces > > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell wrote: > > > > we have the need to create namespaces/authkeys (so that we can run tests in > > parallel); is this supported by the current build env? or would we have to > > launch a separate openwhisk process for each tenant? > > > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana wrote: > > > > > Chetan thanks for thinking on Windows users +1 > > > > > > Just curious why the jar can’t be build in Windows can you log an issue > > > for this? > > > > > > My 2 cents for Windows or Mac I think the best way to build the jar would > > > be inside a docker container. A one liner with a single command should be > > > able to produce the jar on a local workstation or devops pipeline. > > > > > > > > > - Carlos Santana > > > @csantanapr > > > > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra > > > wrote: > > > > > > > > Just to share some more progress. With runnable jar its now possible > > > > to run OpenWhisk standalone even on Windows and execute basic actions. > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10 > > > > > > > > Note you would probably not be able to build the project on windows. > > > > But copying the produced jar would let you run it on Windows. > > > > > > > > Chetan Mehrotra > > > > > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra > > > > wrote: > > > >> > > > >> Hi Nick, > > > >> > > > >> For now just building the PR branch and running following command > > > >> should get you going > > > >> > > > >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar > > > >> > > > >> You can use the default guest and whisk.system credentials to interact > > > >> with it. I am in the process of writing a readme for various options > > > >> exposed. Hope to get it done by Monday > > > >> > > > >> Chetan Mehrotra > > > >> > > > >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell > > > wrote: > > > >>> > > > >>> thanks chetan for doing this! > > > >>> > > > >>> could you provide some example startup sequences, e.g. with sample > > > configs? > > > >>> i'm willing to try this out for our CI/CD pipeline -- i'm sick of 1) > > > >>> waiting 5-7 minutes for openwhisk to start up; and b) fighting ansible > > > >>> versus xenial :) > > > >>> > > > >>> @starpit > > > >>> > > > >>> On Fri, Jun 21, 2019 at 12:44 AM Chetan Mehrotra < > > > chetan.mehro...@gmail.com> > > > >>> wrote: > > > >>> > > > > What's the performance like on startup time > > > > > > It starts in < 5 secs. > > > > > > So far no major blocking issue apart from fetching docker logs on > > > Mac. > > > Current approach of directly reading the log json does not work. So > > > need to have a mac version which uses `docker logs` command to > > > somewhat handle such a case. > > > > > > Chetan Mehrotra > > > > > > > On Thu, Jun 20, 2019 at 9:50 PM James Thomas > > > wrote: > > > > > > > > That is mind-blowingly cool! > > > > > > > > What's the performance like on startup time? > > > > > > > > On Thu, 20 Jun 2019 at 06:14, Carlos Santana > > > wrote: > > > >> > > > >> Genius! > > > >> https://www.adminsub.net/tcp-udp-port-finder/whisker > > > >> > > > >> > > > >> - Carlos Santana > > > >> @csantanapr > > > >> > > > >>> On Jun 19, 2019, at 12:30 PM, David P Grove > > > wrote: > > > >>> > > > >>> WhiskerControl > > >
Re: Re: Backpressure for slow activation storage in Invoker
Okie so we can then aim for adding an optional support for storing activations via a separate service. Currently we also send the activation result on respective controller topic. With this change we would also be sending same activation record on another topic. So we have another choice to make Option B1 - Activations via controller topic Here we avoid a new topic and instead have a service which listen to all controller topics for activation records. However that would be tricky to implement and also tricky to scale out. As scaling out such a service by running multiple copies would not be easy in terms of sharding/partitioning Here the benefit is that we reduce the duplicate writes on Kafka. Option B2 - Introduce a new topic altogether --- We introduce a new topic to which all invokers write the activation records (like the case for user-events). Then implementing a new service to read from a single (possibly partitioned topic) would be easier. My suggestion is to go for B2 for now. Any feedback on that? Chetan Mehrotra On Fri, Jun 21, 2019 at 11:46 PM Rodric Rabbah wrote: > > > Can we handle these in same way as user events? Maybe exactly like user > events, as in use a single service to process both topics. > > good call - the user events already contains much of the activation record > (if not all modulo the logs)? > > -r
