Re: API Gateway routing on Mac

[Rodric Rabbah]

> Lastly the invite bot for your slack channel seems to not be sending
> out invites.

Thanks for alerting us. I checked this out and it is working from what I can 
see. I can send you a direct invite if you like. 

Re: API Gateway routing on Mac

[Rodric Rabbah]

Hi Tom. I’ve used quick start on Mac with api gateway successfully in the past. 
I’ll see if I can reproduce your issue. 

-r

> On Nov 18, 2019, at 11:26 AM, Tom Barber  wrote:
> 
> Just to expand on my deployment question, I appreciate make
> quick-start isn't production ready. I'm just curious if its not API
> gateway ready either for Mac users?
> 
> -- 
> 
> 
> Spicule Limited is registered in England & Wales. Company Number: 
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> 
> 
> 
> 
> All engagements 
> are subject to Spicule Terms and Conditions of Business. This email and its 
> contents are intended solely for the individual to whom it is addressed and 
> may contain information that is confidential, privileged or otherwise 
> protected from disclosure, distributing or copying. Any views or opinions 
> presented in this email are solely those of the author and do not 
> necessarily represent those of Spicule Limited. The company accepts no 
> liability for any damage caused by any virus transmitted by this email. If 
> you have received this message in error, please notify us immediately by 
> reply email before deleting it from your system. Service of legal notice 
> cannot be effected on Spicule Limited by email.

Re: OpenWhisk as a single docker image?

[Rodric Rabbah]

I think we're ready for a new release - at the last community call we set a
goal of pushing out a release vote before the end of November. I expect
we'll take stock today on the community call and should be ready to follow
it up with a vote shortly thereafter.

Given some recent feedback and questions about vagrant and compose, I'd
like to see if we can remove vagrant from the docs (or entirely), and make
sure the standalone controller is front and center on the README/getting
started.

-r


On Wed, Nov 13, 2019 at 9:02 AM Carlos Santana  wrote:

> +1 The jar can be build and made available as a nightly/dev artifact for
> quick development and testing purposes for convenience.
>
> Then provide the one liner command to get up and running, assuming jre and
> docker is present :-)
>
> The jar can’t and should not be distributed as part of an Apache release.
>
> - Carlos Santana
> @csantanapr
>
> > On Nov 13, 2019, at 8:55 AM, Michele Sciabarra 
> wrote:
> >
> > I did some experimentations with the docker in docker and ultimately I
> am convinced it is actually better just to use the jar. I would then
> suggest to distribute at least the precompiled jar instead of requiring to
> compile by himself.
> >
> > --
> >  Michele Sciabarra
> >  mich...@sciabarra.com
> >
> > - Original message -
> > From: Chetan Mehrotra 
> > To: dev@openwhisk.apache.org
> > Subject: Re: OpenWhisk as a single docker image?
> > Date: Wednesday, November 13, 2019 6:00 AM
> >
> > Running standalone logic within a Docker would be tricky as there is
> quite
> > a bit of logic involved to use the right IP for inter docker
> communications
> > specially in Kafka and Api Gateway part.
> >
> > Hence currently we expect that to use standalone the user has at minimum
> > JDK + Docker on the host
> >
> > Chetan Mehrotra
> >
> >
> >> On Tue, Nov 12, 2019 at 10:58 PM Matt Sicker  wrote:
> >>
> >> Docker in Docker has some slight security improvements depending on
> >> your use case, too, compared to just mounting your docker socket into
> >> the running container.
> >>
> >>> On Sat, 9 Nov 2019 at 10:30, Tyson Norris 
> >>> wrote:
> >>>
> >>> I suspect that due to Docker-in-Docker scenario, it will be easier to
> >> use java+jar (+local docker) instead of running the jar in a container.
> >>>
> >>> Today you can start the jar with only java , but you will need a
> >> bunch of parameters (probably different per OS?) to run it in a
> container,
> >> I think.
> >>> Local docker client is switched per OS here
> >>
> https://github.com/apache/openwhisk/blob/231e739373ef681c44b5647a6956d5838a87db2e/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/docker/StandaloneDockerContainerFactory.scala#L37
> >>> I guess this wouldn't apply if running in a container, but it arguably
> >> makes running the jar simpler than running the container IMHO.
> >>> I also suspect you won't get the behavior of launching playground ui to
> >> browser either, which I would miss.
> >>>
> >>> Tyson
> >>>
> >>> On 11/9/19, 5:38 AM, "Michele Sciabarra" 
> wrote:
> >>>
> >>>Wow. I missed those evolutions. So I guess it should not be hard to
> >> package it as a docker image.
> >>>
> >>>To be able to say to people: execute "docker run -p 8080:8080
> >> openwhisk/standalone" and enjoy...
> >>>
> >>>If it is possible I can volounteer to write the dockerfile do
> that...
> >>>
> >>>I have a question: does it use the local docker? Where is the
> >> invoker?
> >>>
> >>>
> >>>--
> >>>  Michele Sciabarra
> >>>  mich...@sciabarra.com
> >>>
> >>>- Original message -
> >>>From: Rodric Rabbah 
> >>>To: dev@openwhisk.apache.org
> >>>Subject: Re: OpenWhisk as a single docker image?
> >>>Date: Saturday, November 09, 2019 2:31 PM
> >>>
> >>>Do you mean the standalone controller?
> >>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwhisk%2Fblob%2Fmaster%2Fcore%2Fstandalone%2FREADME.mddata=02%7C01%7Ctnorris%40adobe.com%7Cfc313c39337a44a5882a08d7651a0cbc%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637089034867332217sdata=OCnWo8R5OfbKLaSQCEeI%2B7pqz0ewp%2BYQGBK2msMoMtc%3Dreserved=0
> >>>
> >>>-r
> >>>
> >>>> On Nov 9, 2019, at 8:18 AM, Michele Sciabarra <
> >> mich...@sciabarra.com> wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>> I remember the discussion about the openwhisk as a single
> >> executable that includes also Kafka. So I wonder: is it now possible to
> run
> >> (for development purposes of course) OpenWhisk as single docker image
> if we
> >> add also couchdb to that one? Because I have an use case where even a
> >> docker-compose can be inconvenient...
> >>>>
> >>>> --
> >>>> Michele Sciabarra
> >>>> mich...@sciabarra.com
> >>>
> >>>
> >>
> >>
> >> --
> >> Matt Sicker 
> >>
>

adding 'updated' field to the API response

[Rodric Rabbah]

Thanks Seonghyun Oh for PR https://github.com/apache/openwhisk/pull/4646
from September which adds the 'updated' field to the API response for whisk
assets.

I have reviewed the PR and approved it, and pending CI passing, intend to
merge it.

-r

Re: ContainerPool buffering changes

[Rodric Rabbah]

Hi Tyson. I’ll have a look.  

-r

> On Nov 12, 2019, at 2:00 PM, Tyson Norris  wrote:
> 
> Seeking someone to review this PR – I’ve been load testing it this week with 
> good success.
> https://github.com/apache/openwhisk/pull/4593
> 
> We can discuss at the call tomorrow if nobody reviews it before then.
> 
> Anyone?
> Thanks
> Tyson
> 
> From: Tyson Norris 
> Date: Thursday, November 7, 2019 at 6:34 AM
> To: "dev@openwhisk.apache.org" 
> Subject: ContainerPool buffering changes
> 
> Hi –
> I have a long outstanding PR to change buffer processing at ContainerPool 
> https://github.com/apache/openwhisk/pull/4593
> The background is that in cases where container scheduling fails, we should 
> not immediately retry scheduling, but rather wait for a resource-affecting 
> event to occur, and then retry. With the previous impl, we saw cases where 
> scheduling would get into a tight loop and crash the invoker.
> 
> Please let me know if you have any concerns?
> 
> Thanks
> Tyson

Re: Providing a namespace field when action is initialized

[Rodric Rabbah]

I haven’t looked at the relevant code recently in the invoker but I reckon all 
the context variables are available at init time. Sending deadline and 
activation id is ok. The deadline has to be sent twice since it will change 
from init to run. 

-r

> On Nov 5, 2019, at 12:44 AM, Seong Hyun Oh  wrote:
> 
> +1
> Yes, It is suitable for my use case.
> 
> I think these environment contexts can be moved to /init
> 
> auth environment
> - api_host (__OW_API_KEY)
> - api_key (__OW_API_HOST)
> 
> environment
> - namespace (__OW_NAMESPACE)
> - action_name (__OW_ACTION_NAME)
> 
> And some environments can not be moved to it.
> 
> not possible
> - (x) activaiton_id (__OW_ACTIVATION_ID)
> - (x) deadline (__OW_DEADLINE)
> 
> 
> 2019년 11월 5일 (화) 오후 2:24, Rodric Rabbah 님이 작성:
> 
>> The namespace is currently provided on /run as part of the environment
>> context. I think the entire context should be moved to /init. This would
>> also address your use case. Wdyt.
>> 
>> -r
>> 
>>> On Nov 5, 2019, at 12:04 AM, Seong Hyun Oh 
>> wrote:
>>> 
>>> Hi, whiskers.
>>> I want to discuss adding a new field `namespace` in the `/init` API [1]
>>> that initializes the action.
>>> 
>>> The action runtime does not know the namespace information when the
>> action
>>> is initialized.
>>> In some cases, this namespace information can be useful.
>>> 
>>> E.g,
>>> - Initializing a monitoring tool such as APM (Application Performance
>>> Management)
>>> - Custom docker actions that require a namespace for initialization
>>> 
>>> If the namespace is not provided at action init time, runtimes that need
>> a
>>> namespace should check the namespace every time when action is run.
>>> 
>>> I'd like to have your views on this proposal
>>> 
>>> {
>>> "value": {
>>>   "namespace": String, <- this `namespace` field will be added
>>>   "name" : String,
>>>   "main" : String,
>>>   "code" : String,
>>>   "binary": Boolean,
>>>   "env": Map[String, String]
>>> }
>>> }
>>> 
>>> Thanks
>>> Seonghyun
>>> 
>>> [1]
>>> 
>> https://github.com/apache/openwhisk/blob/master/docs/actions-new.md#initialization
>> 

Re: Providing a namespace field when action is initialized

[Rodric Rabbah]

The namespace is currently provided on /run as part of the environment context. 
I think the entire context should be moved to /init. This would also address 
your use case. Wdyt.  

-r

> On Nov 5, 2019, at 12:04 AM, Seong Hyun Oh  wrote:
> 
> Hi, whiskers.
> I want to discuss adding a new field `namespace` in the `/init` API [1]
> that initializes the action.
> 
> The action runtime does not know the namespace information when the action
> is initialized.
> In some cases, this namespace information can be useful.
> 
> E.g,
> - Initializing a monitoring tool such as APM (Application Performance
> Management)
> - Custom docker actions that require a namespace for initialization
> 
> If the namespace is not provided at action init time, runtimes that need a
> namespace should check the namespace every time when action is run.
> 
> I'd like to have your views on this proposal
> 
> {
>  "value": {
>"namespace": String, <- this `namespace` field will be added
>"name" : String,
>"main" : String,
>"code" : String,
>"binary": Boolean,
>"env": Map[String, String]
>  }
> }
> 
> Thanks
> Seonghyun
> 
> [1]
> https://github.com/apache/openwhisk/blob/master/docs/actions-new.md#initialization

Remove legacy view code from subjects index.

[Rodric Rabbah]

I opened PR 4638 [1] a little over a month ago to remove legacy subject
indexing in the auth database (which predates a bijective mapping from
namespaces to keys).

This PR introduces a new subjects view (v2) and removes the old view.
Operators can still use the old view because the view is now programmable
at deployment time (from a previous PR).

Sven if you're able to review this PR, your input would be appreciated.
Otherwise, will proceed to merge by silent assent after 48 hours.

[1] https://github.com/apache/openwhisk/pull/4638

-r

Re: Action health checks

[Rodric Rabbah]

as a longer term point to consider, i think the current model of "best
effort at most once" was the wrong design point - if we embraced failure
and just retried (at least once), then failure at this level would lead to
retries which is reasonable.

if we added a third health route or introduced a health check, would we
increase the critical path?

-r

On Tue, Oct 29, 2019 at 12:29 PM David P Grove  wrote:

> Tyson Norris  wrote on 10/28/2019 11:17:50 AM:
> > I'm curious to know what other
> > folks think about "generic active probing from invoker" vs "docker/
> > mesos/k8s specific integrations for reacting to container failures"?
> >
>
> From a pure maintenance and testing perspective I think a single common
> mechanism would be best if we can do it with acceptable runtime overhead.
>
> --dave
>

bump to scala 2.12.10

[Rodric Rabbah]

This PR form 10 days ago bump the scala version to 2.12.10 (from 2.12.9).
If you have concerns about this change please have a look or it will be
merged in 48hrs by silent assent.

https://github.com/apache/openwhisk/pull/4694

-r

Re: Make the formula to calculate the action time limit more configurable

[Rodric Rabbah]

Makes sense to me also.

Docker images are also not capped and always pulled on demand. Have you
considered an alternate and more efficient management strategy that shifts
the pulls to create time perhaps?

-r
On Fri, Oct 18, 2019 at 10:01 AM Steffen Rost  wrote:

> Recently, we observed a considerable amount of forced completion acks on
> our systems. While forced completion acks make sense in some scenarios,
> they cause trouble in our scenario - please see below for details. As a
> band-aid for our scenario, we want to make the forced completion ack
> timeout more configurable.
>
> The completion ack timeout is the timeout within a completion ack must be
> received for an activation. It is calculated based on the action time
> limit. The current formula is:
> (actionTimeLimit.max(TimeLimit.STD_DURATION) * lbConfig.timeoutFactor) +
> 1.minute  (for implementation details please follow the link under [1])
>
> The default timeout factor is 2 which bases on invoker behavior that a
> cold invocation's init duration may be as long as its run duration. Based
> on this formula the calculated completion ack for an action with a timout
> limit of 60 seconds is be 180 seconds.
>
> The motivation behind the completion ack timeout and discarding
> activations from the system that do not complete within that time is to
> not wait "forever" for activations that get lost. This could happen if
> activations were already read and committed from the kafka topic by the
> message feed but their processing is still in flight while at the same
> time the invoker is restarted for whatever reason.
>
> While restarting invokers will rather remain the exception we often have
> the case that image pulls for cold black box invocations take a long time
> and exceed the calculated completion ack timeout for these invocation in
> our environment. By discarding activations that are still being processed
> by an invoker the controllers bookkeeping is invalidated step by step
> because the controller assumes that for each of the discarded invocations
> one invoker slot get freed up while it is not. As consequence the
> controller will make false decisions and what is even worse its
> bookkeeping that is out of sync won't repair by itself but remain in this
> state as long as the workload remains high. Activations have to wait for
> its processing on the chosen invoker as no free slots are available and
> hence will potentially exceed their completion ack timeout and in the end
> being discarded by the controller.
>
> To make a long story short we would like to have the possibility to have
> the constant duration of 1 minute configurable.By increasing the duration
> to an appropriate number and by this the calculated completion ack timeout
> we think we can avoid the forced completion of activations in our system
> for many of the situations we observed in the past.
>
> Please let me know what you think.
>
>
> [1]
>
> https://github.com/apache/openwhisk/blob/81ac503f7efc8ee99ea1a37ef9ec3d6163d96c85/core/controller/src/main/scala/org/apache/openwhisk/core/loadBalancer/CommonLoadBalancer.scala#L86-L104
>
>
> Mit freundlichen Gruessen / Kind regards
> Steffen Rost
>
> --
> IBM Cloud Functions Development
> Phone +49-7031-16-4841 (Fax: -3545)
> E-Mail: sr...@de.ibm.com
>
> --
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Matthias Hartmann -- Geschäftsführung:
> Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>

[Call for Topics] Tech Interchange Call - Wednesday 10/16/19

[Rodric Rabbah]

Hello OpenWhisk community.

I am hosting the Tech Interchange Call on Wednesday October 16th at 10am
Eastern (other times listed below). Call on Zoom:
https://zoom.us/my/asfopenwhisk

Arrive early before the Pizza and Cookies are all gone.
There is also a prize for volunteering to host the next tech call.

If you are interested in sharing something you've been working on, have
some questions you'd like to discuss, or want to raise anything else with
the community please let me know by end of your local business day Tuesday.

Tentative topics:
- Demo the OpenWhisk playground along with the standalone controller
(thanks Chetan)
- Proposal from Bill Zhong https://github.com/apache/openwhisk/pull/4648
- OpenWhisk main repo release (v1.0?)

Day-Time: Wednesday October 16 at 10am Eastern, 7am Pacific, 4pm Central
Europe, 2pm GMT, 10pm Beijing, 11pm Seoul.
Zoom: https://zoom.us/my/asfopenwhisk

-r

Re: [VOTE] Release Apache OpenWhisk Runtime Go (v1.14.0, rc1)

[Rodric Rabbah]

  +1 Approve the release

Release verification checklist for reference:
  [x] Download links are valid.
  [x] Checksums and PGP signatures are valid.
  [x] Source code artifacts have correct names matching the current release.
  [x] LICENSE and NOTICE files are correct for each OpenWhisk repository.
  [x] All files have license headers as specified by OpenWhisk project
policy.
  [x] No compiled archives bundled in source archive.

Re: [DISCUSS] release openwhisk-runtime-go 1.14.0

[Rodric Rabbah]

I suggested in the call yesterday to remove the “loop” in the name. Is that
too much to include in this release?

On Thu, Oct 3, 2019 at 9:43 AM Michele Sciabarra 
wrote:

> Please note that currently all the runtimes build the proxy directly from
> the sources of the master on github.
>
> I did it as a temporary workaround for the "de incubation". If we want to
> make things properly we should release first the golang runtime (without
> knative, no problem for now), then build the other runtimes using a
> released version (that did not exist at the time as the old released one
> was broken by de de-incubation and the redirect was not working)
>
> I can take care of the required PR but please first release the golang and
> provide a stable url for the sources, then I can send PR for all the others
> to use the stable and released version.
>
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>
> - Original message -
> From: David P Grove 
> To: OpenWhisk Dev 
> Subject: [DISCUSS] release openwhisk-runtime-go 1.14.0
> Date: Thursday, October 03, 2019 3:28 PM
>
> I'd like to make a release of the go runtime to enable us to then make
> updated releases of all the action-loop based runtimes (including the rust
> 1.34 runtime which has never been released).
>
> I'd suggest we make this release without waiting to merge the Knative
> support PR (#106) to the go runtime. From the discussion on the technical
> interchange yesterday there may be some iteration needed to normalize
> Knative support across the various runtime projects.
>
> Any objections to proceeding with a 1.14.0 release of openwhisk-runtime-go?
> If not, I can start the process for openwhisk-runtime-go on Friday.  After
> that is done, we can then do the rest of the actionloop based runtimes
> enmasse the second half of next week.
>
> --dave
>

Re: [VOTE] Release Apache OpenWhisk Command-line Interface (CLI), OpenWhisk Client Go, OpenWhisk Wskdeploy (v1.0.0, rc1)

[Rodric Rabbah]

+1 for all three. I checked with rcverify and manual checklist.

-r

On Tue, Sep 17, 2019 at 9:26 AM David P Grove  wrote:

>
>
> Hi,
>
> This is a call to vote on releasing version 1.0.0 release candidate rc1 of
> the following 3 project modules with artifacts built from the Git
> repositories and commit IDs listed below.
>
> * OpenWhisk Command-line Interface (CLI):
> 5ad2291882e1d89ecd8c377dd1ffb1fe1043bf07
>
>
> https://github.com/apache/openwhisk-cli/commits/5ad2291882e1d89ecd8c377dd1ffb1fe1043bf07
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-cli-1.0.0-sources.tar.gz
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-cli-1.0.0-sources.tar.gz.asc
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-cli-1.0.0-sources.tar.gz.sha512
>
> * OpenWhisk Client Go: 716c6f973eb297b39e764938284350050f3d3974
>
>
> https://github.com/apache/openwhisk-client-go/commits/716c6f973eb297b39e764938284350050f3d3974
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-client-go-1.0.0-sources.tar.gz
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-client-go-1.0.0-sources.tar.gz.asc
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-client-go-1.0.0-sources.tar.gz.sha512
>
> * OpenWhisk Wskdeploy: fff873eaf189071e28e9d29deab26a7270027f14
>
>
> https://github.com/apache/openwhisk-wskdeploy/commits/fff873eaf189071e28e9d29deab26a7270027f14
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-wskdeploy-1.0.0-sources.tar.gz
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-wskdeploy-1.0.0-sources.tar.gz.asc
>
>
> https://dist.apache.org/repos/dist/dev/openwhisk/apache-openwhisk-1.0.0-rc1/openwhisk-wskdeploy-1.0.0-sources.tar.gz.sha512
>
> This release is comprised of source code distribution only.
>
> You can use this UNIX script to download the release and verify the
> checklist below:
>
> https://gitbox.apache.org/repos/asf?p=openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=eac6141
>
> Usage:
> curl -s "
>
> https://gitbox.apache.org/repos/asf?p=openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=eac6141
> " -o rcverify.sh
> chmod +x rcverify.sh
> rcverify.sh openwhisk-cli 'OpenWhisk Command-line Interface (CLI)' 1.0.0
> rc1
> rcverify.sh openwhisk-client-go 'OpenWhisk Client Go' 1.0.0 rc1
> rcverify.sh openwhisk-wskdeploy 'OpenWhisk Wskdeploy' 1.0.0 rc1
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> Release verification checklist for reference:
>   [ ] Download links are valid.
>   [ ] Checksums and PGP signatures are valid.
>   [ ] Source code artifacts have correct names matching the current
> release.
>   [ ] LICENSE and NOTICE files are correct for each OpenWhisk repository.
>   [ ] All files have license headers as specified by OpenWhisk project
> policy [1].
>   [ ] No compiled archives bundled in source archive.
>
> This majority vote is open for at least 72 hours.
>
>
> [1]
>
> https://github.com/apache/openwhisk-release/blob/master/docs/license_compliance.md
>

Re: acceptable file types in 'tests' portion of source releases?

[Rodric Rabbah]

For the main repo, the sources from which the zips are made should also be
in the repo.
We can create a build.sh script to generate the zipped artifacts for the
tests (and avoid changing the tests).
I can do that if there are no takers. Maybe create a similar issue for the
main repo with help wanted/good first issue tags.

-r

On Tue, Sep 17, 2019 at 9:17 AM David P Grove  wrote:

>
>
>
> Carlos Santana  wrote on 09/15/2019 06:22:19 PM:
> >
> > Even if want to we should not include in the release tgz anything
> > that is not simple source code files (ie jar,zip,binaryexec)
> >
> > The released tgz can contain files and scripts (also files) that
> > could generate those artifacts (jar, zip, binaryexec)
> >
> > This is my understanding
> >
> > As a receiver of the released tgz to use in a comercial product I
> > would not trust those type of files (jar, zip, binaryexec), but I
> > would be ok to run some commands using the content of the tgz to
> > generate the test features artifacts (jar,zip, binaryexec) that
> > allow me to run the tests.
> >
>
> Makes sense.
>
> To make progress on getting a cli 1.0.0 out, I am going to make a src
> release that simply excludes wskdeploy's integration tests.  I don't have
> the bandwidth to actually fix the underlying problem in wskdeploy test
> suite.  I opened [1] if someone wants to pick it up.
>
> We're going to have the exact same problem with the tests/dat tree of the
> core repo.  Would be great if someone wants to look at dealing with that so
> we can progress on getting a core release out to let us publicize
> standalone.
>
> --dave
>
> [1] https://github.com/apache/openwhisk-wskdeploy/issues/1073
>

Re: Dangers of renaming and removing runtime kinds

[Rodric Rabbah]

I don't think there is actually a distinction between the two paths in
deserialize().  The try path throws the exception inside docReader.read()
whereas in the catch, the exception is deferred to the actual type check
that occurs on lines 64-67. The exceptions should arguably be the same - I
suspect we can eliminate the try/catch (caveat: it's been a while since I
looked at that code carefully).

The reason the deserializer is the way it is, and the order matters, is
that the type of the record is not recorded in the document and so the
deserializer relies on schema matches to deserializes a given document. An
action and a trigger are similar in schema - if you eliminate the exec
property from the former. Perhaps the db interface should address that too
(i.e., record the type in the document since by default there is only one
db for all assets).

-r

On Mon, Sep 16, 2019 at 1:51 PM Sven Lange-Last 
wrote:

> Hello Openwhisk community members,
>
> recently, PR #4390 [1] renamed runtime kind "java" to "java:8". While a
> change like this looks harmless at the first sight, it breaks all existing
> actions of this kind. This may not be important for developers and
> "occasional" usage of Openwhisk - but it affects production deployments.
> Production deployments with existing actions require additional migration
> steps when renaming or removing runtime kinds.
>
> I opened PR #4627 to improve documentation. Said PR also adds
> "documentation" to the pre-defined Openwhisk runtime manifest files to
> make developers aware that renaming or removing runtime kinds can cause
> problems.
>
> There is another area that should be improved - but I need help to better
> understand this area...
>
>
> When trying to create an action with a kind that does not exist, a
> reasonable error message is created:
>
> $ wsk action create kind-does-not-exist tests/dat/actions/hello.js --kind
> nodejs:4
> error: Unable to create action 'kind-does-not-exist': The request content
> was malformed:
> kind 'nodejs:4' not in Set(dotnet:2.2, go:1.11, nodejs:10,
> ballerina:0.990, ruby:2.5, nodejs:18, blackbox, swift:4.2, java:8,
> sequence, nodejs:6, nodejs:12, python:3, python:2, php:7.3) (code
> 33bfb55ce44d1dd0bc6e662c49ea9391)
>
>
> When trying to display an action's metadata which has a kind that does not
> exist, the resulting error message is not helpful at all:
>
> $ wsk action get kind-does-not-exist
> error: Unable to get action 'kind-does-not-exist': Resource by this name
> exists but is not in this collection. (code
> 4761468230c344417fd61cdca5922e52)
>
>
> * My conclusion from looking into controller log's and code is that
> deserialization of the ExecMetaDataBase object fails with a
> DeserializationException [3].
> * This exception fails the "try" block in StoreUtils.deserialize() leading
> to a fall-back read in the "catch" block. This fall-back read seems to
> return a WhiskTrigger instead of a WhiskActionMetaData so that a
> DocumentTypeMismatchException is thrown [4].
>   The resulting message can be found in controller logs: "document type
> class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected
> type class org.apache.openwhisk.core.entity.WhiskActionMetaData.".
> * As a result, getEntity() fails with the misleading error message
> mentioned above and HTTP status code 409 (Conflict).
>
> Can somebody explain why [4] has a fall-back and which scenarios are
> addressed by this?
>
> In our scenario, ExecMetaDataBase should probably throw an
> UnknownRuntimeKindException and StoreUtils.deserialize() should not have a
> fall-back for this exception.
>
>
> [1] https://github.com/apache/openwhisk/pull/4390
> [2] https://github.com/apache/openwhisk/pull/4627
> [3]
>
> https://github.com/apache/openwhisk/blob/2036548e62dbf959d91c2328e86318bd7cfa656f/common/scala/src/main/scala/org/apache/openwhisk/core/entity/Exec.scala#L445-L450
> [4]
>
> https://github.com/apache/openwhisk/blob/2036548e62dbf959d91c2328e86318bd7cfa656f/common/scala/src/main/scala/org/apache/openwhisk/core/database/StoreUtils.scala#L58-L67
> [5]
>
> https://github.com/apache/openwhisk/blob/be1e3a19c02956c9be85023a0bb0ff399c21444d/core/controller/src/main/scala/org/apache/openwhisk/core/controller/ApiUtils.scala#L148-L150
>
>
> Mit freundlichen Grüßen / Regards,
>
> Sven Lange-Last
> Senior Software Engineer
> IBM Cloud Functions
> Apache OpenWhisk
>
>
> E-mail: sven.lange-l...@de.ibm.com
> Find me on:
>
>
> Schoenaicher Str. 220
> Boeblingen, 71032
> Germany
>
>
>
>
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>
>

testing activation polling on/off

[Rodric Rabbah]

When invoking an action, the controller waits on a promise of the result to
complete in one of two ways: active ack (response from the invoker) or from
the activations database. The latter is protected by a deployment flag and
may not be enabled. However our tests did not test for both cases: with
database polling and without.

I opened a PR to address this https://github.com/apache/openwhisk/pull/4623
As a side note, the PR also moves the time the controller waits before it
terminates the HTTP response to a deployment configuration. This has the
added benefit that some tests which took 1 minute each can now run with
custom time limits (which I set to a few seconds).

https://github.com/apache/openwhisk/pull/4623

-r

Re: [DISCUSS}: release "cli group" of projects

[Rodric Rabbah]

+1 for for 1.0

On Fri, Sep 13, 2019 at 10:23 AM David P Grove  wrote:

>
>
> I'd like to make a release of the 3 "cli group" projects:
> openwhisk-client-go, openwhisk-wskdeploy, openwhisk-cli.
>
> The main motivation is to pick up the fix for a bug [1] in wskdeploy, which
> causes the `wsk project` subcommand to crash in some common usage scenarios
> in the 0.10.0 release.
>
> It looks to me like the current master branch is stable with no pending PRs
> that need to be merged.  If I missed something, please comment on this
> thread.
>
> One item for discussion is whether we should number this release as 0.11.0
> or go ahead and call it 1.0.0.   To me it seems like the cli api is fairly
> stable, so going to a 1.x.y numbering seems plausible.  But I don't work on
> the cli tools, so I might be overlooking a reason to stay with a 0.x
> number.
>
> thanks,
>
> --dave
>
> [1] https://github.com/apache/openwhisk-wskdeploy/issues/1050
>

PRs ready to merge

[Rodric Rabbah]

There are at least 4 PRs ready to merge and now have the ready label.

https://github.com/apache/openwhisk/pulls?q=is%3Aopen+is%3Apr+label%3Aready

I know Chetan was generating github digest messages briefly (that was
really useful I thought, thanks for the initiative!) and I'd be happy to
add a summary section to those digests to list PRs that are ready to merge
(passed travis & has approved reviews) so that we can merge ready PRs more
quickly.

For now, I applied the ready label to the PRs I thought passed the test and
will merge them by the end of the week if there are no new comments.

-r

optionally skip kafka/zookeeper pulls

[Rodric Rabbah]

This PR from Bill Zong  removes the explicit
zookeeper pull task and uses the 'pull' attribute on 'docker_container'
instead. See https://github.com/apache/openwhisk/pull/4619.

I will merge this by the end of the week unless there's an objection.

-r

bug in active ack protocol

[Rodric Rabbah]

I opened an issue for a bug in the invoker
https://github.com/apache/openwhisk/issues/4614.

There are 5 calls to send a "ack" from an invoker - 4 of them are
Completion messages and only 1 is an active ack with the activation result.
Two of the calls from InvokerReactive only send a Completion message and no
active acks. This is a bug.

Is there a reason not to send a combined completion/active ack instead of
two messages when the result is ready and the messages can be combined?


-r

Re: Allow decision about action result inclusion in logs on a per call basis

[Rodric Rabbah]

t; 1 line of type action_record and 0 lines of type user_log, I formatted
> the
> one line for better readability:
>
> {"activationId":"68484f5c911d412b884f5c911df12b45",
> "duration":498,
> "end":1567785044285,
>...etc. ...
>// here's the result from above we do not want to be suppressed:
> "message":"{\"error\":\"Initialization has failed due to: SyntaxError:
> Unexpected identifier\\n
>   at NodeActionRunner.init (/nodejsAction/runner.js:79:109)\\n
>   at doInit (/nodejsAction/src/service.js:142:31)\\n
>   at initCode (/nodejsAction/src/service.js:81:24)\\n
>   at /nodejsAction/app.js:69:13\\n
>   at Layer.handle [as handle_request]
> (/node_modules/express/lib/router/layer.js:95:5)\\n
>   at next (/node_modules/express/lib/router/route.js:131:13)\\n
>   at Route.dispatch
> (/node_modules/express/lib/router/route.js:112:3)\\n
>   at Layer.handle [as handle_request]
> (/node_modules/express/lib/router/layer.js:95:5)\\n
>   at /node_modules/express/lib/router/index.js:277:22\\n
>   at Function.process_params
> (/node_modules/express/lib/router/index.js:330:12)\"}",
> "name":"compileError",
> "namespace":"ruediger.ma...@de.ibm.com_MySpaceUS1",
> "path":"ruediger.ma...@de.ibm.com_MySpaceUS1/compileError",
> "response":{
>   "status":"action developer error",
>   "success":false
> },
> "type":"activation_record",
>
>   ... and some more fields...
> }
>
> - We want to achieve that the 'result' field in the activation json is
> not
> suppressed in the error case (success=false) and printed out to the
> action
> log so that the user can find more information in the action log about
> the
> error. But we do not want the 'result' field content be part in the
> action
> log if an activation succeeds (success=true). As already said, this is
> just one example and other strategies for adding/suppressing results
> may
> be applicable that also cannot be solved with a simple flag from
> outside
> (by an annotation or a system configuration flag). The small change we
> request does not change the behavior of OpenWhisk and avoids a
> hard-coded
> strategy.
>
> Requested PR is
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwhisk%2Fpull%2F4604data=02%7C01%7Ctnorris%40adobe.com%7Cf191f79cb3e948a9fcd708d732eeac20%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637033872985899145sdata=W6C6xJEo0ap8aswaRcVJ49Y9KRp38sWKVw5jsqZS1lQ%3Dreserved=0
>
> Thanks, Ruediger
>
>
>
>
> From:   Rodric Rabbah 
> To: dev@openwhisk.apache.org
> Date:   06/09/2019 02:05
> Subject:[EXTERNAL] Re: Allow decision about action result
> inclusion in logs on a per call basis
>
>
>
> I think it?s not clear to me when you say ?log?.
>
> The activation record has a log field and a result field. Are you
> trying
> to add error logs to the result field?
>
> At first I thought you meant you wanted to omit the result field from
> the
> activation record except during errors. But I think I misunderstood.
>
> Maybe an example will be helpful.
>
> -r
>
> > On Sep 5, 2019, at 7:31 PM, Ruediger Maass <
> ruediger.ma...@de.ibm.com>
> wrote:
> >
> > Hi there,
> >
> > may I ask for comments? Annotations for the decision whether results
> > should be added in the action log or not would not fix the problem
> we
> try
> > to solve and is a different, independent question.
> >
> > We want to add error information in the log but not print results in
> the
>
> > good (i.e., success) case. This can only be done if one inspects the
> > activation 'success' field (or the 'error' field in the result) and
> in
> > this case log the result that contains the error information. We
> could
> > hard-code a strategy in `ActivationFileStorage`. But this has the
> > disadvantage that it would change the default behavior of OpenWhisk
> and
> > not everyone might like this and may have different opinions about
> the
> > "right" strategy. Therefore we propose the more flexible way of
> > introducing a flag, let anyone decide on the right strategy, and not
> > hard-coding a strategy. In the default case, there would be no
> beh

factoring out the name of the subjects view

[Rodric Rabbah]

Hello.

At some point in the past, we factored out the couch view names for actions
and activations but not subjects. The subjects view name is hard coded in
the code and not versioned. This makes it difficult to change that view.
The view name should be factored into the db config object.
I created an issue https://github.com/apache/openwhisk/pull/4610
andprepared a patch for the corresponding change for review and comments
https://github.com/apache/openwhisk/pull/4611.

-r

Re: Add namespace field to activation log

[Rodric Rabbah]

Thanks Jiang - I reviewed the change and it looks good to me. I also don't
think it affects compatibility.

-r

On Thu, Sep 5, 2019 at 9:50 PM 蒋鹏程  wrote:

> Dear all:
> ​
>  While I was using DockerToActivationFileLogStore ​to store acitavation
> logs to a separate file and use logstash to process this file, I found it's
> not convenient to send logs to different indices in ElasticSearch using
> namespace
> as a variable, such as send to "openwhisk-%{namespace}", so I added a
> filed "namespace" in this pr:
> https://github.com/apache/openwhisk/pull/4609, it's a small change,
> Please help and comment on the PR.
> ​
> Thanks
> Jiang PengCheng
>

Re: Allow decision about action result inclusion in logs on a per call basis

[Rodric Rabbah]

dex.js:330:12)"
> 
> 
> This kind of information cannot be logged anywhere else and does not 
> contain sensible information. It is very helpful information for the 
> customer to correct the action. So we do not want to suppress this 
> information.
> 
> 
> 
> From:   James Thomas 
> To: dev@openwhisk.apache.org
> Date:   03/09/2019 15:24
> Subject:[EXTERNAL] Re: Allow decision about action result 
> inclusion in logs on a per call basis
> 
> 
> 
> This is a sensible change and I agree with what Rodric has suggested:
> can we make this a per-action annotation?
> 
>> On Tue, 3 Sep 2019 at 13:28, Rodric Rabbah  wrote:
>> 
>>> Action results may contain sensible data that should not be logged.
>> 
>> I interpret this to mean: not stored in the activation record.
>> 
>> If this is what you mean, why not make this a feature controlled per 
> action
>> using an annotation and let the user decide: ok to save the result, not 
> ok
>> to save the result, only save the result on error.
>> 
>> -r
>> 
>> On Tue, Sep 3, 2019 at 8:17 AM Ruediger Maass 
> 
>> wrote:
>> 
>>> Action results may contain sensible data that should not be logged. 
> There
>>> is a system configuration flag (writeResultToFile) that can be used to
>>> switch on or off the result inclusion in logs. But this is a global 
> switch
>>> that holds for all activations. In our case we want to able to decide 
> per
>>> activation whether or not the result should be included in the log or 
> not.
>>> In our special case we want results to be included in case of errors 
> (in
>>> other words, our predicate function for the decision is: 'Does the 
> result
>>> contain an error field?'). But also other decision logic may be
>>> applicable.
>>> 
>>> This PR 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_openwhisk_pull_4604=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=X0kuppcoBkurjGK5dhY1vViAOVizrmBkK-unFKRbRys=Omwt2ZmfdyerpXiU-HbIwPEvr7znKYAA4DFFhJiyk28=3z4nZkA573qZvgihbr448hMHPhnIC4zy7Y8MeuAENU0=
>  
> 
> is a small change
>>> that extends ActivationFileStorage.
>>> 
>>> Please help and comment on the PR.
>>> 
>>> Thank you, Ruediger
>>> 
>>> 
> 
> 
> 
> -- 
> Regards,
> James Thomas
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Can we adjust the time for the community call

[Rodric Rabbah]

Thanks Dominic for bringing this up. +1 from me.
What about alternating times (every two weeks) so other time zones aren't
always so late in the day?

-r

On Wed, Sep 4, 2019 at 11:27 PM Dominic Kim  wrote:

> Dear community members.
>
> I wonder whether we can set up our Tech. Int. meeting 1-hour earlier.
> Currently, we hold the meeting at 3 PM GMT which is 12:00 AM KST/JST.
>
> If we hold the meeting 1-hour earlier, the time changes like this:
>
> - 2 PM GMT.
> - 10 AM US eastern time
> - 10 CST
> - 11 KST/JST
>
> One issue is the US western time would be 7 AM.
> So if anyone lives in that area, I would give up.
>
> But if it's fine for most of the members, I hope we start the meeting
> 1-hour earlier.
> It would help me and other folks living in Korea/Japan to join the meeting
> better.
>
>
> Best regards
> Dominic
>

Re: Allow decision about action result inclusion in logs on a per call basis

[Rodric Rabbah]

> Action results may contain sensible data that should not be logged.

I interpret this to mean: not stored in the activation record.

If this is what you mean, why not make this a feature controlled per action
using an annotation and let the user decide: ok to save the result, not ok
to save the result, only save the result on error.

-r

On Tue, Sep 3, 2019 at 8:17 AM Ruediger Maass 
wrote:

> Action results may contain sensible data that should not be logged. There
> is a system configuration flag (writeResultToFile) that can be used to
> switch on or off the result inclusion in logs. But this is a global switch
> that holds for all activations. In our case we want to able to decide per
> activation whether or not the result should be included in the log or not.
> In our special case we want results to be included in case of errors (in
> other words, our predicate function for the decision is: 'Does the result
> contain an error field?'). But also other decision logic may be
> applicable.
>
> This PR https://github.com/apache/openwhisk/pull/4604 is a small change
> that extends ActivationFileStorage.
>
> Please help and comment on the PR.
>
> Thank you, Ruediger
>
>

Re: removing "projections" on web actions

[Rodric Rabbah]

 > Assuming no one wants to bother with a feature flag for a deprecated
feature, let's set a concrete date on which the PR will be merged.  My
suggestion is December 1, 2019.  That gives ample time for downstream
operators to do their deprecation process.

I don't want to bother with a feature flag. I think the feature as
implemented is unwieldy and kudos to those that did get it to work. Fine
with me to defer it's already been open 4 months.

That said, I have thought that we could provide projection capabilities but
using X- headers (an argument for keeping some of the code around). I am
not sure/convinced this feature in general though is useful.

-r

Re: removing "projections" on web actions

[Rodric Rabbah]

> Though projections on “.http” got removed two years ago, we confusingly
saw A LOT calls
of the form ".http/foo/bar”. Research showed that those customers
are
accessing the “foo/bar” section of the path via the "message.__ow_path"
property in their action code
to be used as additional parameters.

This is necessary if you want to implement more general APIs and the reason
to remove projections from .json etc is so that you can do the same. With
the current behavior you have actions that will behave differently when
accessed with .http and .json and that's a mistake and very confusing.

-r

Re: [VOTE] Release Apache OpenWhisk API Gateway (v0.11.0, rc1)

[Rodric Rabbah]

  [x] +1 Approve the release

Release verification checklist performed manually:
  [x] Download links are valid.
  [x] Checksums and PGP signatures are valid.
  [x] Source code artifacts have correct names matching the current release.
  [x] LICENSE and NOTICE files are correct for each OpenWhisk repository.
  [x] All files have license headers as specified by OpenWhisk project
policy/
  [x] No compiled archives bundled in source archive.

removing "projections" on web actions

[Rodric Rabbah]

This PR https://github.com/apache/openwhisk/pull/4592 removes the
documentation for projecting results of actions via the web invoke path.

This was previously discussed
https://github.com/apache/openwhisk/issues/4352 and there is a pending PR
to also remove the functionality but I thought removing the documentation
is a good first step that should not be controversial.

This feature is cumbersome to use in practice and for meaningful
applications.
Please comment on the PR if you have input otherwise will progress by
silent assent.

-r

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

> Here are a few more examples of "meta" information, other than transaction 
> id.
> - QoS (e.g., to affect scheduling)
> - URL of specific log channel (e.g., to consolidate logs across a chain of 
> invocations)
> - Flow ID (e.g., for tracing)
> - Debug flag (e.g., to enable step-into interactive debugging)

Unlike the transaction id (flow id?), these are all properties that don’t exist 
today in the control plane. 

-r

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

Note the transaction id is already user visible. It’s sent in the response 
headers. As long as we are clear and document that there’s no api for querying 
by this id at this time, the risk of confusion is low imo.

-r

> On Aug 21, 2019, at 11:16 AM, Martin Henke  wrote:
> 
> Chetan,
> 
> from an operational point of view I have some fear that we will confuse the 
> user by making the transaction id visible as a second id besides the 
> activation id. 
> Some will certainly use it to fetch activation records and fail, which will 
> lead to questions.
> Any thoughts from your side ?
> 
> Regards,
> Martin
> 
> 
>> On 20. Aug 2019, at 12:32, Chetan Mehrotra  wrote:
>> 
>> I created a separate thread to discuss how to store such metadata related
>> to activation.
>> 
>> Current open PR #4586 only enables exposing the transactionId to env. It
>> does not make any attempt to store the transactionId currently. Once we
>> decide how such data should be stored then I can open PR for  the same
>> 
>> Chetan Mehrotra
>> 
>> 
>>> On Mon, Aug 19, 2019 at 8:47 AM Rodric Rabbah  wrote:
>>> 
>>> Yes indeed. Your pr already open I think is fine as is.
>>> 
>>> -r
>>> 
>>> On Aug 19, 2019, at 11:36 AM, Chetan Mehrotra 
>>> wrote:
>>> 
>>>>> That’s true. Time for api/v2...
>>>> 
>>>> This is now becoming a rabbit hole! What option should we use without
>>> going
>>>> for v2?
>>>> 
>>>> 1. Introduce a new "meta" sub document
>>>> 2. OR Change annotations to flat map while storing but transform that to
>>>> array based structure while returning to client
>>>> 
>>>> Chetan Mehrotra
>>>> 
>>>> 
>>>>> On Mon, Aug 19, 2019 at 7:15 AM Rodric Rabbah  wrote:
>>>>> 
>>>>> 
>>>>>> However changing them now would cause compatibility
>>>>>> issue with various tooling out there which may be interpreting the
>>>>>> annotation per current design
>>>>> 
>>>>> That’s true. Time for api/v2... 
>>> 
> 

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

Yes indeed. Your pr already open I think is fine as is. 

-r

On Aug 19, 2019, at 11:36 AM, Chetan Mehrotra  wrote:

>> That’s true. Time for api/v2...
> 
> This is now becoming a rabbit hole! What option should we use without going
> for v2?
> 
> 1. Introduce a new "meta" sub document
> 2. OR Change annotations to flat map while storing but transform that to
> array based structure while returning to client
> 
> Chetan Mehrotra
> 
> 
>> On Mon, Aug 19, 2019 at 7:15 AM Rodric Rabbah  wrote:
>> 
>> 
>>> However changing them now would cause compatibility
>>> issue with various tooling out there which may be interpreting the
>>> annotation per current design
>> 
>> That’s true. Time for api/v2... 

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

> However changing them now would cause compatibility
> issue with various tooling out there which may be interpreting the
> annotation per current design

That’s true. Time for api/v2... 

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

What if we make annotations a dictionary? 

-r

On Aug 19, 2019, at 9:54 AM, Chetan Mehrotra  wrote:

>> I second the idea of recording the request/transaction id as part of the 
>> activation record. We don’t currently do that,  but we do have a caused-by 
>> annotation for composite activations. I suggest another annotation which is 
>> the transaction id.
> 
> There are few other cases where I felt a need to record some "meta"
> ids in activation
> 
> 1. While integrating with Lambda or other such compute stack I would
> like to save the responseId from the backend system in activation so
> as to enable fetching logs later based on the responseId from previous
> call.
> 2. In Mesos or k8s setup one may want to record the Mesos taskId, k8s
> podId for correlating with some system logs later
> 
> Should we consider it as yet another annotation or add new sub
> document `meta` and make it a property there?
> 
> If we make it an annotation then it becomes tricky to index this
> property in db due to the document structure. For couchdb you can do
> it by extracting the key when generating the view but for other db's
> we typically have to add a computed field at time of persistence.
> 
> Another option would be to introduce a new sub structure say `meta`
> where we store such ids as flat keys
> 
> "meta" : {
>"transactionId" : "xxx",
>"podId" : "ow_xxx"
>}
> 
> Such an structure would be easier to index compared to one based on 
> annotations
> 
> Chetan Mehrotra
> 
>{
>"activationId": "e40c9340aea340f58c9340aea320f5e9",
>"annotations": [
>{
>"key": "causedBy",
>"value": "sequence"
>},
>{
>"key": "kind",
>"value": "nodejs:10"
>},
>{
>"key": "timeout",
>"value": false
>},
>{
>"key": "limits",
>"value": {
>"concurrency": 200,
>    "logs": 10,
>"memory": 256,
>"timeout": 6
>}
>}
>],
>"cause": "f3380aaeca7d4791b80aaeca7d5791f6",
>"duration": 5015,
>"end": 1562344247086,
>"entityType": "activation",
>"response": { ... },
>"start": 1562344242071,
> 
>}
> 
>> On Mon, Aug 19, 2019 at 5:34 AM Rodric Rabbah  wrote:
>> 
>> I second the idea of recording the request/transaction id as part of the 
>> activation record. We don’t currently do that,  but we do have a caused-by 
>> annotation for composite activations. I suggest another annotation which is 
>> the transaction id.
>> 
>> Chetan’s pr is straightforward and I think achieves the goal of linking 
>> parents to children for external tracing.
>> 
>> -r
>> 
>>> On Aug 19, 2019, at 5:02 AM, Erez Hadad  wrote:
>>> 
>>> Hi folks,
>>> 
>>> My two cents: recall my "shared context" presentation about using a
>>> semi-transparent execution context that is shared across multiple
>>> consequent invocations. Similar to the transaction id being discussed.
>>> https://cwiki.apache.org/confluence/download/attachments/74689638/Shared%20Context%20in%20OpenWhisk%20Invocations.pptx?api=v2
>>> 
>>> Specifically, you may want to consider the implementation options in slide
>>> 11:
>>> 1. One particular option of interest is to embed the shared context id ( =
>>> transaction id) in the activation id. When the new activation id is
>>> generated, the transaction id can be taken from the caller's activation id
>>> and embedded in the new activation id. This way the activation id remains
>>> unique per invocation but the transaction id can be extracted from it if
>>> needed, manually or via a client library (e.g., extending the OW js
>>> library). Kind of a minimal change.
>>> 2. Another possibly valuable option is to have the transaction id used as
>>> a key for retrieving data (e.g., key/value) from a 3rd-party fast service,
>>> such as redis. This can be left open for the user of the transaction id or
>>> provide a 

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

I second the idea of recording the request/transaction id as part of the 
activation record. We don’t currently do that,  but we do have a caused-by 
annotation for composite activations. I suggest another annotation which is the 
transaction id. 

Chetan’s pr is straightforward and I think achieves the goal of linking parents 
to children for external tracing. 

-r

> On Aug 19, 2019, at 5:02 AM, Erez Hadad  wrote:
> 
> Hi folks,
> 
> My two cents: recall my "shared context" presentation about using a 
> semi-transparent execution context that is shared across multiple 
> consequent invocations. Similar to the transaction id being discussed. 
> https://cwiki.apache.org/confluence/download/attachments/74689638/Shared%20Context%20in%20OpenWhisk%20Invocations.pptx?api=v2
> 
> Specifically, you may want to consider the implementation options in slide 
> 11:
> 1. One particular option of interest is to embed the shared context id ( = 
> transaction id) in the activation id. When the new activation id is 
> generated, the transaction id can be taken from the caller's activation id 
> and embedded in the new activation id. This way the activation id remains 
> unique per invocation but the transaction id can be extracted from it if 
> needed, manually or via a client library (e.g., extending the OW js 
> library). Kind of a minimal change. 
> 2. Another possibly valuable option is to have the transaction id used as 
> a key for retrieving data (e.g., key/value) from a 3rd-party fast service, 
> such as redis. This can be left open for the user of the transaction id or 
> provide a reference implementation in the client library. Such data can 
> also be pre-fetched before the invocation starts (e.g., from CouchDB / 
> Redis), but again, at the cost of a higher code change.
> 
> Regards,
> -- Erez
> 
> 
> 
> 
> 
> 
> From:   Rodric Rabbah 
> To: dev@openwhisk.apache.org
> Cc: tyson.nor...@gmail.com
> Date:   17/08/2019 03:36
> Subject:[EXTERNAL] Re: Passing TransactionId as part of action 
> invocation
> 
> 
> 
> Of course if the transaction id is the same as the activation id (of the 
> composite action) the solutions are comparable. 
> 
> The downside of using transaction id is that we have no APIs to query by 
> it today, and it?s not recorded in the activation metadata. So while it?s 
> useful for external tracing (no objections to doing it), I think from an 
> openwhisk API point of view we still have a gap. 
> 
> -r
> 
> On Aug 16, 2019, at 4:58 PM, Chetan Mehrotra  
> wrote:
> 
>>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
>>> header (using the existing transaction id/X-Request-Id), the parent is 
> not
>>> needed?
>> 
>> Thats what I counting on as we just need to correlate calls made for a
>> given flow corelated with time to get a sequence of flow.
>> 
>>> - expose the transaction id to runtime container
>>> - propagate the transaction id in requests initiated from runtime
>>> container/controller
>> 
>> Makes sense. Would open a ticket capturing these changes then
>> Chetan Mehrotra
>> 
>>> On Fri, Aug 16, 2019 at 7:44 AM Tyson Norris  
> wrote:
>>> 
>>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
>>> header (using the existing transaction id/X-Request-Id), the parent is 
> not
>>> needed? i.e. there may be 2 parts to this effort:
>>> - expose the transaction id to runtime container
>>> - propagate the transaction id in requests initiated from runtime
>>> container/controller
>>> 
>>> 
>>> 
>>>> On Thu, Aug 15, 2019 at 10:38 AM Rodric Rabbah  
> wrote:
>>>> 
>>>> In general yes but I think generally do you need the transaction id or 
> the
>>>> parent id for an activation?
>>>> 
>>>> This issue is relevant - 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_openwhisk_issues_3083=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=Oo9B0p_tCCWIIum5GpjjqA=RVvqHH3EKTdA9cNFz3n2dPeXV_wFUOU01I5vxvAqt8Q=eZzttWaoeE0WFe_MndFO1O6K1wA9hZbc-BKbRZa6kDM=
>  
> .
>>>> I also recall in the early days of the composer, we wanted a way to 
> query
>>>> parent/child activations but this requires new couch views and we 
> didn't
>>>> pursue it.
>>>> 
>>>> 
>>>> 
>>>> On Thu, Aug 15, 2019 at 1:20 PM Chetan Mehrotra 
> >>>> 
>>>> wrote:
>>>> 
>>>>> Currently we pass the `activation_id` as part of `/run` call to any
>>>>> a

init parameters

[Rodric Rabbah]

Is anyone up for reviewing this PR 
https://github.com/apache/openwhisk/pull/4559 which adds the ability to 
separate parameters into init time vs run time?

It’s been open for a while with no feedback. 
Previously discussed here [1] and implemented accordingly. 

-r

[1] 
https://lists.apache.org/thread.html/7e03e6fc965d972882f4c76e34b2e9aa28579a0c3004d63d546e2611@%3Cdev.openwhisk.apache.org%3E

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

Of course if the transaction id is the same as the activation id (of the 
composite action) the solutions are comparable. 

The downside of using transaction id is that we have no APIs to query by it 
today, and it’s not recorded in the activation metadata. So while it’s useful 
for external tracing (no objections to doing it), I think from an openwhisk API 
point of view we still have a gap. 

-r

On Aug 16, 2019, at 4:58 PM, Chetan Mehrotra  wrote:

>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
>> header (using the existing transaction id/X-Request-Id), the parent is not
>> needed?
> 
> Thats what I counting on as we just need to correlate calls made for a
> given flow corelated with time to get a sequence of flow.
> 
>> - expose the transaction id to runtime container
>> - propagate the transaction id in requests initiated from runtime
>> container/controller
> 
> Makes sense. Would open a ticket capturing these changes then
> Chetan Mehrotra
> 
>> On Fri, Aug 16, 2019 at 7:44 AM Tyson Norris  wrote:
>> 
>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
>> header (using the existing transaction id/X-Request-Id), the parent is not
>> needed? i.e. there may be 2 parts to this effort:
>> - expose the transaction id to runtime container
>> - propagate the transaction id in requests initiated from runtime
>> container/controller
>> 
>> 
>> 
>>> On Thu, Aug 15, 2019 at 10:38 AM Rodric Rabbah  wrote:
>>> 
>>> In general yes but I think generally do you need the transaction id or the
>>> parent id for an activation?
>>> 
>>> This issue is relevant - https://github.com/apache/openwhisk/issues/3083.
>>> I also recall in the early days of the composer, we wanted a way to query
>>> parent/child activations but this requires new couch views and we didn't
>>> pursue it.
>>> 
>>> 
>>> 
>>> On Thu, Aug 15, 2019 at 1:20 PM Chetan Mehrotra >>> 
>>> wrote:
>>> 
>>>> Currently we pass the `activation_id` as part of `/run` call to any
>>>> action runtime [1]. Would it be fine to also pass the `TransactionId`
>>>> such that it can be accessed by action code?
>>>> 
>>>> One usecase of this would be to enable tracing a sequence/composition
>>>> by linking all activations which are part of same transaction in
>>>> epsagon [2]
>>>> 
>>>> Chetan Mehrotra
>>>> [1]
>>>> 
>>> https://github.com/apache/openwhisk/blob/master/docs/actions-new.md#activation
>>>> [2]
>>>> 
>>> https://epsagon.com/blog/epsagon-makes-troubleshooting-apache-openwhisk-a-snap/
>>>> 
>>> 

Re: Passing TransactionId as part of action invocation

[Rodric Rabbah]

In general yes but I think generally do you need the transaction id or the
parent id for an activation?

This issue is relevant - https://github.com/apache/openwhisk/issues/3083.
I also recall in the early days of the composer, we wanted a way to query
parent/child activations but this requires new couch views and we didn't
pursue it.



On Thu, Aug 15, 2019 at 1:20 PM Chetan Mehrotra 
wrote:

> Currently we pass the `activation_id` as part of `/run` call to any
> action runtime [1]. Would it be fine to also pass the `TransactionId`
> such that it can be accessed by action code?
>
> One usecase of this would be to enable tracing a sequence/composition
> by linking all activations which are part of same transaction in
> epsagon [2]
>
> Chetan Mehrotra
> [1]
> https://github.com/apache/openwhisk/blob/master/docs/actions-new.md#activation
> [2]
> https://epsagon.com/blog/epsagon-makes-troubleshooting-apache-openwhisk-a-snap/
>

Re: OpenWhisk swag at ApacheCon?

[Rodric Rabbah]

Thanks Matt for the heads up. The wiki says "The following list of projects
have at least one presentation on the schedule or will have some of their
community present and would like some stickers available at the Apache
booth;" I don't think if we have any openwhisk presentations at the
conference. I'll probe a bit.

I'm happy to have some stickers or other swag (mug/shirt) sent to you.

-r

On Wed, Aug 7, 2019 at 8:03 PM Matt Ryan  wrote:

> Hi,
>
> I wonder if anyone on the PMC could reach out to the community to get
> OpenWhisk swag at ApacheCon?  I'd definitely pick up some.
>
> There was an email about this recently; relevant part included below:
>
>
> 1) get swag on RedBubble https://www.redbubble.com/people/comdev --don't
> see your project listed? Find you logo at http://apache.org/logos/ and
> request it be added by sending email to d...@community.apache.org
>
> 2) get stickers --if you'd like us to have your project stickers at the ASF
> Booth (or if you'd like some for a future event), please  request via the
> ComDev wiki
> https://cwiki.apache.org/confluence/display/COMDEV/ApacheCon+NA+2019 or
> email d...@community.apache.org
>
>
> -MR
>

Re: Video + Notes posted from today's OW Tech. Interchange meeting on 2019-08-07

[Rodric Rabbah]

wskdebug is very cool! will be trying it out.
and congrats Michele for the book.
thanks Neeraj for volunteering to host the next call!

-r



On Wed, Aug 7, 2019 at 12:41 PM Matt Rutkowski 
wrote:

> Thanks Chetan for hosting!
>
> A very full and informative call including:
> - wskdebug tool demo and discussion of contribution with Hot code reload
> and browser LiveReload + IDE debugging with NodeJS and Java actions.
> - O'Reilly book on Apache OpenWhisk authored by Michele Sciabarra who
> walked us through contents
>
> Notes:
> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-08-07+OW+Tech+Interchange+-+Meeting+Notes
> Video: https://youtu.be/Qtsi8VFm0uY
>
> I believe I heard Neeraj volunteer to be the next moderator for the 21st...
>
> Cheers,
> Matt
>

Re: housecleaning of openwhisk git repos

[Rodric Rabbah]

Arching makes the repo read only and indicates it is no longer maintained. 

https://help.github.com/en/articles/archiving-a-github-repository

-r

> On Aug 5, 2019, at 5:15 PM, Priti Desai  wrote:
> 
> 
> 
>> On 2019/08/05 16:51:33, James Thomas  wrote: 
>> Looking over the list, here's a few thoughts from me on the "TO DECIDE" 
>> list...
>> 
>> - incubator-openwhisk-external-resources: This (popular) list of OW
>> content can probably be archived. I've stop maintaining it.
>> - incubator-openwhisk-workshop: I'm not sure this even works
>> anymore[1] - happy to archive.
>> - incubator-openwhisk-tutorial: Don't think this works? Should archive.
>> - incubator-openwhisk-client-swift: Unmaintained (wouldn't assume it
>> still works...) - Archive.
>> - incubator-openwhisk-slackinvite: Still being used but not updated - 
>> archive?
>> 
>> [1] - 
>> https://github.com/apache/incubator-openwhisk-workshop/blob/master/exercises/setting_up_cli/exercise.js#L19
>> 
>>> On Thu, 1 Aug 2019 at 21:21, David P Grove  wrote:
>>> 
>>> Appended is Chetan's list of repos broken into suggested KEEP (28 repos),
>>> NEED DECISION (10 repos) , and ARCHIVE (14 repos) sections.
>>> 
>>> I plan to open a ticket for infra to rename (remove incubator-) for the 28
>>> repos on the KEEP list tomorrow.  Will hold off on acting on the other 24
>>> for now.
>>> 
>>> --dave
>>> 
>>> KEEP
>>> 1 | incubator-openwhisk| 2019-07-24
>>> 2 | incubator-openwhisk-website| 2019-07-23
>>> 3 | incubator-openwhisk-cli| 2019-07-22
>>> 4 | incubator-openwhisk-runtime-swift  | 2019-07-22
>>> 5 | incubator-openwhisk-runtime-go | 2019-07-20
>>> 6 | incubator-openwhisk-runtime-nodejs | 2019-07-18
>>> 7 | incubator-openwhisk-runtime-rust   | 2019-07-18
>>> 8 | incubator-openwhisk-deploy-kube| 2019-07-18
>>> 9 | incubator-openwhisk-client-js  | 2019-07-16
>>> 10| incubator-openwhisk-runtime-python | 2019-07-12
>>> 11| incubator-openwhisk-devtools   | 2019-07-11
>>> 12| incubator-openwhisk-utilities  | 2019-07-10
>>> 13| incubator-openwhisk-release| 2019-07-09
>>> 14| incubator-openwhisk-catalog| 2019-07-08
>>> 15| incubator-openwhisk-wskdeploy  | 2019-07-05
>>> 16| incubator-openwhisk-composer   | 2019-07-04
>>> 17| incubator-openwhisk-runtime-php| 2019-07-03
>>> 18| incubator-openwhisk-runtime-java   | 2019-07-03
>>> 19| incubator-openwhisk-apigateway | 2019-07-01
>>> 20| incubator-openwhisk-package-alarms | 2019-06-29
>>> 21| incubator-openwhisk-package-cloudant   | 2019-06-29
>>> 22| incubator-openwhisk-package-kafka  | 2019-06-29
>>> 23| incubator-openwhisk-runtime-ballerina  | 2019-06-29
>>> 24| incubator-openwhisk-runtime-dotnet | 2019-06-29
>>> 25| incubator-openwhisk-runtime-ruby   | 2019-06-29
>>> 26| incubator-openwhisk-runtime-docker | 2019-06-29
>>> 27| incubator-openwhisk-client-go  | 2019-06-29
>>> 28| incubator-openwhisk-pluggable-provider | 2019-06-24
>>> 
>>> 
>>> NEED DECISION:
>>> 29| incubator-openwhisk-test   | 2019-05-31
>>> 31| incubator-openwhisk-composer-python| 2019-03-20
>>> 32| incubator-openwhisk-external-resources | 2019-02-10
>>> 33| incubator-openwhisk-workshop   | 2019-01-24
>>> 34| incubator-openwhisk-package-pushnotifications  | 2019-01-08
>>> 35| incubator-openwhisk-package-deploy | 2018-11-07
>>> 36| incubator-openwhisk-tutorial   | 2018-07-30
>>> 37| incubator-openwhisk-client-swift   | 2018-05-09
>>> 38| incubator-openwhisk-deploy-mesos   | 2018-04-16
>>> 39| incubator-openwhisk-slackinvite| 2017-12-29
>>> 
>>> 
>>> ARCHIVE:
>>> 30| incubator-openwhisk-deploy-openshift   | 2019-05-20
>>> 40| incubator-openwhisk-vscode | 2017-11-03
>>> 41| incubator-openwhisk-package-rss| 2017-09-27
>>> 42| incubator-openwhisk-xcode  | 2017-08-21
>>> 43| incubator-openwhisk-package-template   | 2017-08-17
>>> 44| incubator-openwhisk-debugger   | 2017-07-20
>>> 45| incubator-openwhisk-sample-matos   | 2017-07-14
>>> 46| incubator-openwhisk-playground | 2017-07-12
>>> 47| incubator-openwhisk-podspecs   | 2017-07-12
>>> 48| incubator-openwhisk-client-python  | 2017-07-10
>>> 49| incubator-openwhisk-package-jira   | 2017-07-10
>>> 50| 

Re: Publishing a pre-release version of Standalone OpenWhisk

[Rodric Rabbah]

It would certainly help in getting others to try it. 

Since the nightly builds are not releases and must not be labeled as releases I 
think the issue you’re alluding to is not the same. 

-r

> On Aug 1, 2019, at 4:57 AM, Chetan Mehrotra  wrote:
> 
> Would it be useful to publish a pre-release version of Standalone
> OpenWhisk to simplify trials of this mode by users until we do a
> proper release of core repo?
> 
> Currently I have a task open (#4525) to publish nightly release to
> Github org [1]. However based on some recent discussions I am not sure
> if its fine to push such nightly releases in automated way via Travis.
> 
> Another option can be to push a snapshot build to Apache Maven repo
> [3] via some Jenkins job. However this work may take some time as we
> would need to upgrade Gradle to 5.x which has better Maven release
> support and which we need for supporting JDK 11
> 
> So was thinking to manually publish a pre-release build at [2] based
> on local build and eventually support publishing snapshot builds to
> Apache Maven repo
> 
> Thoughts on what approach to take here?
> 
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/issues/4525
> [2] https://github.com/apache/incubator-openwhisk/releases
> [3] https://repository.apache.org/

Re: Try out the api gateway support in standalone server (#4571)

[Rodric Rabbah]

Hi Chetan,

this is fantastic... I followed your instructions here as is even though I
noticed the gw PR was merged already and in theory I should be able to drop
the image override.

the system came up cleanly, and as relayed on Slack, I think it would be
useful to hoist the messages that gw is ready up to where the system checks
are first printed so they aren't lost in all the output that follows which
for the most part, users should be able to ignore.

i tried creating an API as documented in [2] but hit a snag. The controller
received this request from the gateway:

[#tid_bEVIJzQ...] GET //localhost:3233/api/v1/web/guest/default/hello.json

but this doesn't match any routes in the controller and the API returns
with a 404. The prefix "//localhost:3233/" needs to be omitted, I'm not yet
sure where it's coming from. I suspect you'll know right away.

These are the logs from the gw for the failed request:

==> /var/log/api-gateway/access.log <==
172.17.0.1 - remote_user="-" [29/Jul/2019:14:50:45 +]
request_method="GET"
request_uri="/api/23bc46b1-71f6-4ed5-8c54-816aa4f8c502/hello/world"
request_protocol="HTTP/1.1" status=404 bbs=96 rl=130 rt=0.008 hr="-"
ua="curl/7.54.0" xfwdf="-" upadd="192.168.65.2:3233" upstat=404 uprt=0.007
tenantId="23bc46b1-71f6-4ed5-8c54-816aa4f8c502"
tenantNamespace="23bc46b1-71f6-4ed5-8c54-816aa4f8c502" tenantInstance=""
requestHeaders="" requestBody="" responseHeaders="" responseBody=""
apiId="8a68d0a9-a024-4411-9233-7ba1d9099b1a" apiKey="-"
analyticsUri="/hello/world" reqid="zvL6RbV0qtiZ0BakfDGudINV8Gzv1Ocl"

==> /var/log/api-gateway/gateway_error.log <==
2019/07/29 14:50:45 [info] 35#0: *21 client 172.17.0.1 closed keepalive
connection
2019/07/29 14:50:48 Executing sync cmd: echo ''
sync stdout | ''
2019/07/29 14:50:48 done


-r



On Mon, Jul 29, 2019 at 7:53 AM Chetan Mehrotra 
wrote:

> Hi Team,
>
> As discussed in last Tech Interchange call support is now being added
> to launch the Api Gateway with Standalone Server [1]. With this it
> should be possible to direct new OpenWhisk users to use the Standalone
> server to try out various features provided by OpenWhisk
>
> So far tests look fine and would now like to seek some feedback from
> community on this feature. For this it would be helpful if you can try
> out a beta build and see if it works as expected
>
> # Fetch custom docker build of apigateway which has
> apache/incubator-openwhisk-apigateway#347 fixed
> $ docker pull chetanmeh/apigateway
>
> # Download the openwhisk-standalone jar
> $ wget
> https://github.com/chetanmeh/incubator-openwhisk/releases/download/0.11/openwhisk-standalone.jar
> $ java -Dwhisk.standalone.api-gateway.image=chetanmeh/apigateway -jar
> openwhisk-standalone.jar --api-gw
>
> Post this you can use the steps as mentioned in [2] to check out the
> gateway support.
>
> Please share any feedback on #4571 or this mail thread
>
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/pull/4571
> [2]
> https://github.com/apache/incubator-openwhisk/blob/master/docs/apigateway.md
>

Re: Video+notes posted from today's tech. int. call

[Rodric Rabbah]

Thanks Matt for the notes. Sorry I couldn’t make it today.

I’ll go through the repo list from Dave and Chetan for my suggested list of 
repos to retire. I’ll do that tomorrow.

-r

> On Jul 24, 2019, at 1:13 PM, Matt Rutkowski  wrote:
> 
> First post-grad. meeting!
> 
> video: https://youtu.be/x_MtqtYyLQw
> notes: 
> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-07-24+OW+Tech+Interchange+-+Meeting+Notes
> 
> Chetan kindly agreed to host the next call sched. for Aug. 7th

Re: add reason parameter for wskadmin tool

[Rodric Rabbah]

Thanks Steffen for adding this feature. I agree it's useful.

Could you add a test for the change?

In terms of divergence between wskadmin and wskadmin-next, I propose that
we separately deprecate wskadmin and all new features are added to
wskadmin-next.

-r


On Wed, Jul 24, 2019 at 6:28 AM Steffen Rost  wrote:

> I would like to propose the possibility to store in addition the reason
> for setting a certain limit using the wskadmin tool. This will help others
> to understand why limits are stored for a certain namespace when
> revisiting them.
>
> I already opened a pull request to add a single limit parameter to the
> wskadmin tool [1]. This PR also discuss the implementation of a more
> sophisticated audit log. Please check the PR for details.
>
> Please let me know what you think.
>
> [1] https://github.com/apache/incubator-openwhisk/pull/4564
>
>
>
> Mit freundlichen Gruessen / Kind regards
> Steffen Rost
>
> --
> IBM Cloud Functions Development
> Phone +49-7031-16-4841 (Fax: -3545)
> E-Mail: sr...@de.ibm.com
>
> --
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Matthias Hartmann -- Geschäftsführung:
> Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>

Re: stricter scancode: now rejecting minified license headers

[Rodric Rabbah]

The scancode configuration is excluding the tests/data/actions directory.
Since we're not bundling tests in the releases, I suppose that was OK in
the past. But we should be including tests, and in any case we should
remove this exclusion and review the scan code exclusions to make sure
they're not too loose.

# Exclude from incubator-openwhisk


# Jenkins/test generated reports


# Test data.


# Generated binary artifacts


tests/build/reports
tests/dat/actions
docs/images
bin



On Wed, Jul 24, 2019 at 11:25 AM Rodric Rabbah  wrote:

> We've got a gap somewhere.
>
> https://github.com/apache/incubator-openwhisk/blob/master/tests/dat/actions/argsPrint.js
> and several other js files in the tests use the short license.
>
> -r
>
> On Wed, Jul 24, 2019 at 11:04 AM David P Grove  wrote:
>
>>
>>
>> I just merged [1] which enforces the stricter license header conventions
>> in
>> scancode (only use full header, not minified) that we discussed on the dev
>> list about a month ago.   Matt has gone through a large number of repos
>> and
>> updated all the license headers, so hopefully we are ready for the
>> stricter
>> rules.
>>
>> --dave
>>
>> [1] https://github.com/apache/incubator-openwhisk-utilities/pull/65
>>
>

Re: stricter scancode: now rejecting minified license headers

[Rodric Rabbah]

We've got a gap somewhere.
https://github.com/apache/incubator-openwhisk/blob/master/tests/dat/actions/argsPrint.js
and several other js files in the tests use the short license.

-r

On Wed, Jul 24, 2019 at 11:04 AM David P Grove  wrote:

>
>
> I just merged [1] which enforces the stricter license header conventions in
> scancode (only use full header, not minified) that we discussed on the dev
> list about a month ago.   Matt has gone through a large number of repos and
> updated all the license headers, so hopefully we are ready for the stricter
> rules.
>
> --dave
>
> [1] https://github.com/apache/incubator-openwhisk-utilities/pull/65
>

Re: A direction to take in a new component, "the scheduler"

[Rodric Rabbah]

We should put together a roadmap that shows the evolution and all presents a 
coherent view of all the moving parts. 

-r

> On Jul 18, 2019, at 8:52 AM, Markus Thömmes  wrote:
> 
> Hi,
> 
> this sounds a lot like the "OpenWhisk 2.0" discussion we had in an older
> thread where I proposed an architectural overhaul. See this thread for
> context:
> https://lists.apache.org/thread.html/d19306dd976138a153f48d32c5a55f2853e4b8ff405fc46f7260e905@%3Cdev.openwhisk.apache.org%3E
> 
> There have been strong opinions in the past against overhauling.
> 
> Cheers,
> Markus
> 
> Am Do., 18. Juli 2019 um 13:42 Uhr schrieb Dominic Kim > :
> 
>> Recently I discussed the direction and a safe way to add a new component
>> with Sven Lange-Last.
>> Let me share the discussion results.
>> More feedbacks and critique are welcomed.
>> 
>> Since the implementation includes a breaking architectural change, it comes
>> with a risk.
>> It must not break any existing downstream system as well as the upstream
>> pipelines.
>> So all implementations should be disabled by default along with proper
>> switches.
>> 
>> The new change would require many iterations to become stable enough for a
>> production system.
>> It would be better to have a new separate (openwhisk) CI pipeline for the
>> changes.
>> All unit/system tests will only be executed on the new CI pipeline.
>> 
>> I will defer to Sven, he may add more comments.
>> 
>> Best regards
>> Dominic
>> 

Re: A direction to take in a new component, "the scheduler"

[Rodric Rabbah]

> It must not break any existing downstream system as well as the upstream 
> pipelines. So all implementations should be disabled by default along with 
> proper
> switches.

Agreed. There is a feature flag class that Chetan introduced when I introduced 
a breaking change (sorry) a few months ago that should be useful here.

> It would be better to have a new separate (openwhisk) CI pipeline for the
> changes.

Do you mean a new Jenkins child which toggles on the related switches? This 
makes sense. 

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

I opened issue https://github.com/apache/incubator-openwhisk/issues/4558
and corresponding PR https://github.com/apache/incubator-openwhisk/pull/4559
.
Additional feedback and critique welcomed.

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

I implemented both, the "convention" and the parameters designated as init
through the API. The latter wasn't much more over the former and is
backward compatible. This can be introduced incrementally into the main
repo/REST API, then the runtimes (I added it to Node but not the others),
and eventually the CLI and other tooling to introduce a "-e".

> - this is also a bigger convention change for action developers (read
from env or context instead of the single args object given to main)

We have no mechanism today for developers to use env vars with their
actions. This new capability actually leads to nicer code -- one prime
example is the use of 'npm openwhisk' which can now be lifted to a top
level constant that's automatically initialized, or using libraries such as
sequelize in a similar way.

So while it would require developers to take advantage of this, it doesn't
break any existing code.

On Thu, Jun 27, 2019 at 1:25 PM Tyson Norris 
wrote:

> BTW, circling back on "can it be done just using -p in current form": I
> still like the idea that:
> - action configured params are different the user specified params (and we
> should only pass them on init, not run)
> - this is also a bigger convention change for action developers (read from
> env or context instead of the single args object given to main)
>
> But - *adding* the -e, instead of revising the meaning of -p does makes
> this a backwards compatible change, so that's a good thing.
>
> Thanks
> Tyson
>
>
>

Welcome new PPMC members

[Rodric Rabbah]

It is my pleasure to share that the OpenWhisk PPMC voted to invite Rob
Allen and Michele Sciabarra as new PPMC members based on their ongoing and
valuable contributions to the project. They have both accepted the
invitation. Rob and Michele were already project committers.

Please join me in welcoming them to this new role on the project!

-r

Re: Updating our contributions guide

[Rodric Rabbah]

Thanks Matt. One of the improvements I thought I’d make is to explain exactly 
this: when a cla is required and when we can accept improvements to 
documentation, typos, minors fixes etc. without. 

-r

> On Jul 13, 2019, at 5:07 PM, Matt Sicker  wrote:
> 
> I've looked around for some existing guidelines around CLA
> requirements, and so far I've found this:
> 
> https://www.apache.org/licenses/contributor-agreements.html#clas
> 
> This relates eventually to the provenance of source code hosted at the ASF:
> 
> https://www.apache.org/foundation/license-faq.html#provenance
> 
> So basically, the general idea I've seen is that small, trivial
> changes do not require an ICLA, but anything non-trivial should
> request one in order to establish provenance of the code over time.
> Remember, safe, business-friendly licensing of all our software is the
> key point to address here, so anything we do here should align with
> that. Since trivial contributions are not usually covered by
> copyright, there's no need to establish more formalities around them.
> Larger ones would retain their own new copyright, and the ICLA is
> there to help ensure the contributions are made along the same rules
> as the ALv2.
> 
>> On Tue, 9 Jul 2019 at 03:06, Rodric Rabbah  wrote:
>> 
>> It was pointed out that our contributing guide sets a higher bar than other 
>> Apache projects by requiring an ICLA.
>> 
>> Looking at some of the other successful Apache projects they do a better job 
>> explaining how to contribute, and all the ways someone can be contributor, 
>> and how to open a PR, testing and how reviewing works.
>> 
>> Should we revise the guidelines and contributions doc? Is anyone else 
>> interested in helping out on these docs?
>> 
>> -r
> 
> 
> 
> -- 
> Matt Sicker 

Re: devtools and make-quickstart

[Rodric Rabbah]

> I also like the standalone JAR. Should we consider adding to that the API
Management features made available through OW GW, or for that we should
keep the docker-compose route ?

I did get the API gateway to run (in the container) with standalone
openwhisk. I ran the ansible playbook (wskdev apigw) to get it up and
going. It doesn't work with sls (serverless framework because of the DNS
resolution issue I've posted about in the past) but will work with wsk.

-r

devtools and make-quickstart

[Rodric Rabbah]

this issue was opened against devtools and raises an important point:
https://github.com/apache/incubator-openwhisk-devtools/issues/273

"On a separate note, will these builds eventually be versioned or will they
continue to be tagged in a backwards incompatible way? I am trying to
create a build process that is deterministic and currently I am unable to
achieve this."

Given that we're pointing developer to make quick start still as the way to
startup and some of the dependence in this build on docker latest (now
nightly), I think we should pin all the dependent images etc and make a
release available.

An existential question is whether we should use the standalone controller
which is faster to startup and adapt our docs accordingly.

Thoughts?

-r

devtools update for mighty docker tag

[Rodric Rabbah]

Hi

This PR https://github.com/apache/incubator-openwhisk-devtools/pull/270 I think 
fixes the docker compose quick start since the docker tags change. Can I get 
help with a review/merge?

I tested make quickstart for myself. We are getting several people reporting 
the failures. 

-r

Re: rationale for wskdeploy Docker image

[Rodric Rabbah]

I think we don’t need it but defer to those who worked on it. 

I’d like to also ask at the risk of broadening the original question: 

- should we fold the repository into the cli? 
- or is there a good reason to keep it as a separate deploy tool independent of 
the cli? 

-r

> On Jul 3, 2019, at 4:23 PM, David P Grove  wrote:
> 
> 
> Do we really need to be publishing a wskdeploy image to dockerhub?
> 
> I'm not understanding why this image needs to be public (it appears to
> perhaps be an image for building wskdeploy?).
> 
> thanks,
> 
> --dave
> 

Updating our contributions guide

[Rodric Rabbah]

It was pointed out that our contributing guide sets a higher bar than other 
Apache projects by requiring an ICLA. 

Looking at some of the other successful Apache projects they do a better job 
explaining how to contribute, and all the ways someone can be contributor, and 
how to open a PR, testing and how reviewing works.

Should we revise the guidelines and contributions doc? Is anyone else 
interested in helping out on these docs?

-r

Re: Fine-grained permission

[Rodric Rabbah]

Thanks Dominic for bringing up this feature. I know PR 4058 has been sitting 
for a while. Does it make sense to accept it (I will review it again it’s been 
a while since I posted my comments last) before going on to do more of the 
permissions work?

We should capture the dev list discussion in an issue. 

-r

> On Jul 6, 2019, at 1:49 AM, Dominic Kim  wrote:
> 
> Thank you for the good point, Martin.
> I will take it into account.
> 
> Best regards
> Dominic
> 
> 
> 2019년 7월 5일 (금) 오후 10:10, Martin Henke 님이 작성:
> 
>> Dominic,
>> 
>> I would very much like to have this change introduced in a backward
>> compatible fashion.
>> 
>> In Detail:
>> The default permissions should either keep the same behaviour as with the
>> current code
>> or better be configurable via Ansible in a way that he system behaves the
>> same as before.
>> If you decide for a schema change, the code should be handle existing
>> actions with the existing behaviour.
>> 
>> With being backward compatible there would be no need to introduce this
>> change in a new EntitlementProvider implementation but can (and should)
>> be placed in the (default) LocalEntitlement provider to have consistent
>> code in the default configuration for Action creation (adding the access
>> rights) and entitlement checking (reading the access rights).
>> 
>> Thank you for driving this forward.,
>> Martin
>> 
>> 
>> 
>>> On 4. Jul 2019, at 17:12, Dominic Kim  wrote:
>>> 
>>> Thank you for the feedback, James.
>>> 
>>> I am thinking of implementing it with annotations.
>>> For example, we would add a new annotation reserved by the system.
>>> When an action is created, the default permission will be specified via
>> the
>>> annotation.
>>> 
>>> When any request comes, a new EntitlementProvider will look for the
>>> annotation and reject the request based on it.
>>> 
>>> This is to minimize the action schema change.
>>> But if it's ok to bear with the schema change, I am ok with adding one
>> more
>>> field in an action other than annotations or parameters.
>>> 
>>> 2019년 7월 4일 (목) 오후 10:07, James Thomas 님이 작성:
>>> 
 Protecting accidental overwritting or deletion of actions would be a
>> great
 feature. I like the suggestion and approach of using Unix-style
 permissions.
 
> On Thu, 4 Jul 2019 at 07:25, Dominic Kim  wrote:
> 
> Recently I discussed this:
> https://github.com/apache/incubator-openwhisk/pull/4058 with my
> colleagues.
> That PR is to add a feature to protect actions from deletion by
>> mistake.
> That is a good suggestion and I think we can also include more
 generalized
> way to handle the issue.
> 
> For example, what we can expect about permission are as follows.
> 
> 1. Action protection.
> 2. Hide codes from the shared package.
> 
> I am a bit faint but IIRC, Rodric suggested linux-like permission
> management.
> 
> Regarding number 1, we can achieve it with the permission, "Read / not
> Write / Execute".
> And with regared to number 2, we can also achieve it with the
>> permission,
> "not Read / not Write (this is the default of shared package action) /
> Execute".
> 
> If we apply linux-like permission to these cases, we can have two
 different
> permission flags, one for owners, the other for users of shared
>> packages.
> Then actions can have permission information such as "71" or "51".
> So "71" would mean the owner of an action can do "read/write/execute"
>> it
> but the one who uses the shared action would be able to do "not
>> read/not
> write/execute".
> "51" would mean the owner can do "read/not write/execute".
> 
> There might be more cases, but I believe we can deal with them in the
 same
> way.
> Any feedback or idea on this would be appreciated.
> 
> Thanks
> Best regards,
> Dominic
> 
 
 
 --
 Regards,
 James Thomas
 
>> 
>> 

API gateway Invalid URI "undefined/tenants"

[Rodric Rabbah]

Several openwhisk users recently have hit a bug using serverless framework
(sls) and openwhisk to deploy a function/api. I've hit the same bug when
using the standalone controller + api gateway.

The symptom is failure to create an API and this error:
Failed to deploy API Gateway route due to error: POST
https://:31001/api/v1/web/whisk.system/apimgmt/createApi.http
Returned HTTP 502 (Bad Gateway) --> "API creation failure: Unable to
configure the API Gateway: "Invalid URI \"undefined/tenants\"""

The issue is documented here
https://github.com/serverless/serverless-openwhisk/issues/103#issuecomment-503732524
and the workaround suggested there worked for me. It is to add

APIGW_ACCESS_TOKEN=APIGW_ACCESS_TOKEN

to the .wskprops file. This shouldn't be necessary - maybe someone else
knows why the workaround works or how to fix it properly in the gateway or
the route management packages.

-r

Re: Re: OpenWhisk invoker overloads - "Rescheduling Run message"

[Rodric Rabbah]

> Instead of resending the message to the pool immediately, it just waits
in the runbuffer, and the runbuffer is processed in reaction to any
potential change in resources: NeedWork, ContainerRemoved, etc. This may
add delay to any buffered message(s), but seems to avoid the catastrophic
crash in our systems.

This makes sense since the rescheduling is really an indication of
something going badly at the container level from my recollection. A better
solution might be to reschedule the request to another invoker for some
fairness criteria (but not easily guaranteed with the current architecture).

(This is testing my memory but...) we used to see these in a previous
incarnation of the scheduler as a precursor to docker daemon going out to
lunch. (Markus might remember better.)

-r

Re: api gateway with standalone controller

[Rodric Rabbah]

Thanks Dragos.

The resolver contents are:

bash-4.4# cat  "/etc/api-gateway/conf.d/includes/resolvers.conf"
resolver 192.168.65.1;


I'm trying to run the API gateway along with the standalone controller. I
started the gateway with "wskdev apigw", and using
http://host.docker.internal:3233 as my openwhisk API host for the wsk CLi.

-r


On Thu, Jun 27, 2019 at 11:28 AM Dascalita Dragos 
wrote:

> Rodric, are you running a custom api gateway container ?
> The lines at [1] should fix the issue for you.
>
> NGINX has its own internal DNS resolution for performance considerations.
> I'm curios what the content of
> "/etc/api-gateway/conf.d/includes/resolvers.conf"
>  is inside the apigateway container.
>
> Also, in case you use a custom configuration file for OW, it's possible
> that the config doesn't include "resolvers.conf" and hence the DNS
> resolution doesn't work ?
>
> [1] -
>
> https://github.com/apache/incubator-openwhisk-apigateway/blob/master/init.sh#L58-L59
>
> On Wed, Jun 26, 2019 at 5:28 PM Rodric Rabbah  wrote:
>
> > I tried to use the api gateway with the standalone controller and ran
> into
> > a few issues which I'll document in the relevant docs soon. I'm now
> hitting
> > an issue where the api gateway cannot resolve the host for routing a
> > request.
> >
> > I opened
> > https://github.com/apache/incubator-openwhisk-apigateway/issues/345
> > to summarize the findings. I created an API successfully which when I
> curl
> > fails in the gateway because it can't resolve the upstream host.
> >
> > I can nslookup/ping the stanalone controller from the api gateay
> container,
> > and I can successfully the webaction URL from the api gateway container
> as
> > well.
> >
> > How does the API gateway resolve a hostname and where is that code?
> Thanks
> > for the help.
> >
> > -r
> >
>

Re: Adding Github app (Depend-a-bot) to Client JS SDK repo?

[Rodric Rabbah]

Hey James, I don't know the answer but can offer this anecdotal evidence. I
attempted to add github app, and requested access for our apache repo over
a week ago and nothing has happened in terms of approval. So I suspect
without INFRA's help, the request falls on the floor.

-r

On Mon, Jul 1, 2019 at 9:50 AM James Thomas  wrote:

> Hello Whiskers.
>
> If I want to add a Github app to one of our repos - do I just need to open
> a JIRA ticket with Infra? Does anyone know if there are any rules about
> what apps are allowed?
>
> I'm continuing work[1] on updating the project dependencies for the Client
> JS SDK, which had not been updated for a long time. I want to make sure
> this doesn't happen again and add a Github app to send PRs each time a
> major dependency upgrade is available using Depend-a-bot:
> https://github.com/marketplace/dependabot-preview
>
> Does anyone have any issues with this?
>
> [1] - https://github.com/apache/incubator-openwhisk-client-js/pull/180
> --
> Regards,
> James Thomas
>

Re: [VOTE] Release Apache OpenWhisk Catalog (v0.10.0-incubating, rc1)

[Rodric Rabbah]

+1

  [x] Download links are valid.
  [x] Checksums and PGP signatures are valid.
  [x] DISCLAIMER is included.
  [x] Source code artifacts have correct names matching the current release.
  [x] LICENSE and NOTICE files are correct for each OpenWhisk repository.
  [x] All files have license headers as specified by OpenWhisk project
policy [1].
  [x] No compiled archives bundled in source archive.

On Sun, Jun 30, 2019 at 3:34 PM David P Grove  wrote:

>
>
> Hi,
>
> This is a call to vote on releasing version 0.10.0-incubating release
> candidate rc1 of the following project module with artifacts built from the
> Git repositories and commit IDs listed below.
>
> * OpenWhisk Catalog: cb8d5c73
>   https://github.com/apache/incubator-openwhisk-catalog/commits/cb8d5c73
>
>
> https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz
>
>
> https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz.asc
>
>
> https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz.sha512
>
> This release is comprised of source code distribution only.
>
> You can use this UNIX script to download the release and verify the
> checklist below:
>
> https://gitbox.apache.org/repos/asf?p=incubator-openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=HEAD
>
> Usage:
> curl -s "
>
> https://gitbox.apache.org/repos/asf?p=incubator-openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=HEAD
> " -o rcverify.sh
> chmod +x rcverify.sh
> rcverify.sh openwhisk-catalog 'OpenWhisk Catalog' 0.10.0-incubating rc1
>
> Please vote to approve this release:
>
>   [ ] +1 Approve the release
>   [ ]  0 Don't care
>   [ ] -1 Don't release, because ...
>
> Release verification checklist for reference:
>   [ ] Download links are valid.
>   [ ] Checksums and PGP signatures are valid.
>   [ ] DISCLAIMER is included.
>   [ ] Source code artifacts have correct names matching the current
> release.
>   [ ] LICENSE and NOTICE files are correct for each OpenWhisk repository.
>   [ ] All files have license headers as specified by OpenWhisk project
> policy [1].
>   [ ] No compiled archives bundled in source archive.
>
> This majority vote is open for at least 72 hours.
>
>
> [1]
>
> https://github.com/apache/incubator-openwhisk-release/blob/master/docs/license_compliance.md
>

log stripping for standalone controller

[Rodric Rabbah]

I opened this PR https://github.com/apache/incubator-openwhisk-cli/pull/444
to allow activation log stripping to work with the standalone controller.
The notable change is the regex for recognizing the timestamp and stream
identifier.

The PR also turns on unit cli test which weren't previously running in
Travis.

-r

Re: [DISCUSS] - prepare to release OpenWhisk catalog

[Rodric Rabbah]

> We can either (a) wait out a release cycle of clientgo/cli/wskdeploy to get a 
> fixed cli or (b) leave the bash installers in for this release and cleanup 
> after the next cli release.

I would do it later then. 

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

Thanks for the added feedback - keep it coming!

Tyson is right in that we shouldn't let implementation concerns affect a
proper design and sacrifcie the experience.

So in line with his concerns, is there a desire to facilitate environment
variables at all (user specified values, not system/context ones)?

If so, then having thought about it some more, annotating every parameter
could be done as suggested say with a -e vs a -p.

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

The use of env vars wouldn't be an issue with intra-container concurrency
if they're immutable, right? The more specific issue that arises today
which I think is your primary concern, are the __OW_ system provided
environment variables which mutate with each invocation of main. Is that
right?

If so, then I think there are only two issues:
1. do we introduce a context object (did you see my other dev list mail?)
2. logging

i really think my issue is orthogonal to these concerns - it's a
convenience feature for the developer.  An action can already export
environment variables today from main.

-r

On Wed, Jun 26, 2019 at 8:44 PM Tyson Norris 
wrote:

> Sorry, what I meant was: accumulating issues around the "main" function,
> which are:
> - context
> - use of env vars
> - your issue: separating user provided params from developer-provided
> params
> - completely separate, but worth noting: logging (and env vars) in the
> face of concurrency
>
> On 6/26/19, 5:29 PM, "Rodric Rabbah"  wrote:
>
> > There are some accumulating issues around init and run.
>
> Which issues are these?
>
> -r
>
>
>

Re: exporting activation arguments to the environment

[Rodric Rabbah]

> There are some accumulating issues around init and run.

Which issues are these?

-r

api gateway with standalone controller

[Rodric Rabbah]

I tried to use the api gateway with the standalone controller and ran into
a few issues which I'll document in the relevant docs soon. I'm now hitting
an issue where the api gateway cannot resolve the host for routing a
request.

I opened https://github.com/apache/incubator-openwhisk-apigateway/issues/345
to summarize the findings. I created an API successfully which when I curl
fails in the gateway because it can't resolve the upstream host.

I can nslookup/ping the stanalone controller from the api gateay container,
and I can successfully the webaction URL from the api gateway container as
well.

How does the API gateway resolve a hostname and where is that code? Thanks
for the help.

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

By the way, an existential question is whether we should facilitate
environment variables at all for actions (a user can still do whatever
they like).
To Rob's point about separating parameters, this could be achieved
through environment variables, or context object (to tie the two dev
threads together).

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

> Sorry, it still seems controversial to me, not sure how others feel?

That's why we discuss on the dev list :D Thanks for the feedback so far.

> can you confirm this is decided based on the case of the parameter name?

Indeed, we need some rule to then partition the parameter list. Using the
convention that the env var starts with a capital letter is one. Other
conventions are plausible.

> adding a '-e' flag that specifically does "set these environment
variables"

Sure - but this increases the complexity of implementation significantly
for not a lot of gain. To add a -e, we'd need to modify the schema for
actions. For example, we could add annotation for each parameter name to be
treated as an environment variable using the existing annotations, and use
these annotations as the criteria. We could create a new field in the
actions object to hold the parameters (a schema change). We could annotate
each parameter (also a schema change).

Since a developer already controls the names of their parameters today,
they have complete control over this partitioning.

If we're open to schema changes, then we can explore a cleaner
implementation but an incremental approach that at least makes the feature
available incrementally would also make sense since making a schema change
is a lot more invasive, coupled with a few changes needed at the invoker
level plus all the runtimes.

-r


On Wed, Jun 26, 2019 at 2:07 PM Tyson Norris 
wrote:

> Sorry, it still seems controversial to me, not sure how others feel?
>
> To be clear, when you added "-a  partition-arguments true", the result is
> 2 things:
> 1. some of the -p args are now treated different than others - can you
> confirm this is decided based on the case of the parameter name?
> 2. init receives these params (which sounds good to me).
>
> Regardless of opting in to this behavior, having action-configured
> parameters referenced differently based on the name of the param seems
> bad.  I understand there are some useful conventions defining these as env
> vars, but my point is that this doesn't seem at all like an explicit
> choice. I think an explicit choice would be more like adding a '-e' flag
> that specifically does "set these environment variables", instead of
> overloading the '-p' flag with a convention based on the name of the
> variable.
>
> Thanks
> Tyson
>
>
> On 6/26/19, 10:43 AM, "Rodric Rabbah"  wrote:
>
> Maybe this got missed, but here's how I conceived of this. I'll use
> wsk CLI
> commands, I think it makes it obvious.
>
> wsk action create myAction code.js -p MY_ENV true -p some_param false
> -a
> partition-arguments true
>
> The annotation (partition-arguments) makes it explicit for the
> developer to
> control whether "main" receives the arguments as they do today, which
> is
> this object
> { MY_ENV: true, some_param: false}, or when the annotation is true, {
> some_param: false} and process.env.MY_ENV is set to true.
>
> I don't think there's anything confusing about this in that the
> developer
> has decided what variables to export to the environment, and is making
> an
> explicit choice.
>
> Environment variables on a number of platforms are restricted to those
> at
> consist of words that start with capital letter (AWS, Netlify as two
> prime
> examples).
>
> The alternative, today, requires a function to export any variables
> from
> "main" to the environment. So it would explicitly export MY_ENV to the
> environment. The change we're discussing frees the programmer from
> having
> to do that.
>
> The change to the runtime proxies would be 1. accept an additional
> value on
> /init and export all the properties it contains to the environment.
>
> Before I address the POST invoke issue, I'd like to make sure my
> explanation is clearer and if this is still controversial.
>
> -r
>
>
> On Wed, Jun 26, 2019 at 1:21 PM Tyson Norris  >
> wrote:
>
> > Are you saying one is exported to environment, during init, based on
> > parameter name being UPPER case? Forgetting use of env vars for a
> minute,
> > this seems confusing to treat parameters different based on names. I
> would
> > rather see either a) all action-configured params sent to init only,
> and
> > never to run or b) all action-configured params sent to run as
> context
> > object.
> >
> > What the runtime does at init (use env vars or not) can be different
> per
> > runtime, but in the action-configured parameter case I don't see any
> > problem with setting env vars, except that

Re: exporting activation arguments to the environment

[Rodric Rabbah]

Maybe this got missed, but here's how I conceived of this. I'll use wsk CLI
commands, I think it makes it obvious.

wsk action create myAction code.js -p MY_ENV true -p some_param false -a
partition-arguments true

The annotation (partition-arguments) makes it explicit for the developer to
control whether "main" receives the arguments as they do today, which is
this object
{ MY_ENV: true, some_param: false}, or when the annotation is true, {
some_param: false} and process.env.MY_ENV is set to true.

I don't think there's anything confusing about this in that the developer
has decided what variables to export to the environment, and is making an
explicit choice.

Environment variables on a number of platforms are restricted to those at
consist of words that start with capital letter (AWS, Netlify as two prime
examples).

The alternative, today, requires a function to export any variables from
"main" to the environment. So it would explicitly export MY_ENV to the
environment. The change we're discussing frees the programmer from having
to do that.

The change to the runtime proxies would be 1. accept an additional value on
/init and export all the properties it contains to the environment.

Before I address the POST invoke issue, I'd like to make sure my
explanation is clearer and if this is still controversial.

-r


On Wed, Jun 26, 2019 at 1:21 PM Tyson Norris 
wrote:

> Are you saying one is exported to environment, during init, based on
> parameter name being UPPER case? Forgetting use of env vars for a minute,
> this seems confusing to treat parameters different based on names. I would
> rather see either a) all action-configured params sent to init only, and
> never to run or b) all action-configured params sent to run as context
> object.
>
> What the runtime does at init (use env vars or not) can be different per
> runtime, but in the action-configured parameter case I don't see any
> problem with setting env vars, except that there seems to be a convention
> in some cases that allows invoking clients to "override" these values using
> POST parameters at invocation time. This also seems confusing but could
> also be enforced differently by various runtimes, although ideally I would
> rather see the convention change to: action-configured parameters are
> always sent to init, and always visible to run, regardless of what client
> sends as execution parameters.
>
> Thanks
> Tyson
>
>
> On 6/25/19, 3:32 PM, "Rodric Rabbah"  wrote:
>
> Context and Knative I view as orthogonal.
>
> That is, for the context object, it is another way of encapsulating
> arguments. It doesn’t export variable to the process environment.
>
> You can provide an action with both environment variables, arguments
> to main, and a context object. They are orthogonal.
>
> For the context object, the distinction that was necessary from
> previous discussions was related to separating intra container concurrent
> executions. If the system-provided context is exported to the environment
> as it today the values clobber each other. For this, the context object
> would make sense.
>
> I’m simply talking about two parameters wsk ... “-p a A” and “-p B b”
> say where one becomes exported to the environment as B=b and the other is
> passed to the action as ({a:A}).
>
> I’m going to set the knative discussion aside because I think it’s a
> distraction. With knative you can bind environment variables to the
> container. As you would with any other container.
>
> I think it’s too simplistic to say knative has a single endpoint.
> After all there are readiness probes and possible pre/post start hooks that
> operators may have to deal with. Init can be viewed as the readiness probe.
>
> Fundamentally I believe the actor model is much better aligned with
> the reactive programming model for functions so this will tend toward a
> completely different discussion in my view.
>
> The reason my proposal sets the environment variables at init time is
> that’s how env vars work;  they exist before you start you process. While
> they don’t need to be immutable, it makes sense to test them as such.
>
> For webaction parameters that one would export to an environment, they
> are already immutable and cannot be overridden. So really you would not use
> them for anything that varies per activation.
>
> The view here is that you can export global (immutable) variables to
> the action. This makes it easier to take existing code and containers which
> might use env vars and use them almost off the shelf.
>
> -r
>
> > On Jun 25, 2019, at 6:07 PM, Tyson Norris 
> wrote:
> >
> > I had to read this several times, but have some suggestions.

Re: Notes & Video posted from today's Tech Int. call

[Rodric Rabbah]

James Thomas hosted today. Thank you James. And thanks Matt for the notes and 
agreeing to moderate the next one. 

I thought we agreed to skip July 10 so the next one is July 24.

I can volunteer for the one after that (August) if no one else volunteers. I 
will miss the next call. 

-r

> On Jun 26, 2019, at 12:51 PM, Matt Rutkowski  wrote:
> 
> 
> Notes: 
> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-06-26+OW+Tech+Interchange+-+Meeting+Notes
> Video: https://youtu.be/HkIzJXlTw68
> 
> Thanks Rodric for hosting... Matt is up next call for July 10th iteration.

Re: [ACTION REQUIRED] Nominations for PMC Chair for TLP graduation resolution

[Rodric Rabbah]

Dave and I conferred and we've agreed he's in a better position today to
chair the PMC. I'm happy to be in a able to learn from him in the inaugural
year, assuming a successful petition to graduate.

-r

On Tue, Jun 25, 2019 at 5:04 PM Matt Rutkowski 
wrote:

>
> This PMC Chair nominations thread is closed.  Will reach out to both Dave
> and Rodric, as both appeared to defer to each other, to see if either wants
> to step back and perhaps we can reduce the need for a competitive VOTE (and
> have just an affirmation VOTE as this may be asked for) and move onto
> seeing if we want to create a yearly nomination/rotation system where
> either Dave or Rodric serve for the first year...
>
> Thanks everyone for participating.
>
> -Matt
>
> >
> > This thread is intended to solicit nominations (self or on-behalf of
> another) to begin the process of selecting a candidate for election (i.e.,
> via a subsequent vote thread) to this position.  This thread will be  kept
> open until Tuesday at 1PM US EDT which is one week from now (and greater
> than the 72-hour required min. for any VOTE).
> >
> > After all nominations are accounted for, and after the nominees
> acknowledge their interest in fulfilling the role of PMC Chair, we can then
> discuss the process of holding an official vote.
> >
> > I encourage anyone interested in leading the project forward, regardless
> of current status or contribution area, to express their intentions here
> knowing that you will have the support of our amazing Community and PMC who
> are full of excitement around embracing new technologies and potential
> integrations.
> >
>

context object to action

[Rodric Rabbah]

In an earlier email discussion [1] Tyson noted the previously
discussed idea of using a context object per activation in lieu of
environment variables which are currently exported by the
system/runtime per activation. This was previously discussed in [2]
and is necessary to separate intra-container concurrency where
otherwise environment variables would clobber each other.

I have an implementation of the nodejs proxy which allows one to
specify the main handlers with a context object.
exports.main = (args, context) => { }

I've only done it for nodejs and don't currently have plans to support
other runtimes. I'm happy to open a PR against the nodejs repo for
this. The patch does not export the __OW context objects and instead
uses methods/getters on the context object. In doing this mapping, I
adopted AWS Lambda's model for the method names, so that you can take
an AWS lambda for nodejs and run it in OpenWhisk.

Of course there are some caveats. Two primarily:
- the first is that if a AWS lambda that makes heavy use of AWS
specific features provided to it, they will not have analogs.
- AWS is somehow detecting that the node event loop has emptied and
can block the lambda (or return) depending on the event loop status. I
haven't replicated this behavior in openwhisk.

In any case, the Lambda compatibility for nodejs is tangentially
related to the point I started with: it already provides a context
object which can address Tyson's issue [2].

-r

[1] 
https://lists.apache.org/thread.html/d0791b9397f1fcd2a0e47290381fd94bc9a76744ff4837288be027c1@%3Cdev.openwhisk.apache.org%3E

[2] https://github.com/apache/incubator-openwhisk/issues/4020

-r

Re: exporting activation arguments to the environment

[Rodric Rabbah]

Context and Knative I view as orthogonal. 

That is, for the context object, it is another way of encapsulating arguments. 
It doesn’t export variable to the process environment. 

You can provide an action with both environment variables, arguments to main, 
and a context object. They are orthogonal.

For the context object, the distinction that was necessary from previous 
discussions was related to separating intra container concurrent executions. If 
the system-provided context is exported to the environment as it today the 
values clobber each other. For this, the context object would make sense. 

I’m simply talking about two parameters wsk ... “-p a A” and “-p B b”  say 
where one becomes exported to the environment as B=b and the other is passed to 
the action as ({a:A}). 

I’m going to set the knative discussion aside because I think it’s a 
distraction. With knative you can bind environment variables to the container. 
As you would with any other container.

I think it’s too simplistic to say knative has a single endpoint. After all 
there are readiness probes and possible pre/post start hooks that operators may 
have to deal with. Init can be viewed as the readiness probe.

Fundamentally I believe the actor model is much better aligned with the 
reactive programming model for functions so this will tend toward a completely 
different discussion in my view.

The reason my proposal sets the environment variables at init time is that’s 
how env vars work;  they exist before you start you process. While they don’t 
need to be immutable, it makes sense to test them as such. 

For webaction parameters that one would export to an environment, they are 
already immutable and cannot be overridden. So really you would not use them 
for anything that varies per activation.

The view here is that you can export global (immutable) variables to the 
action. This makes it easier to take existing code and containers which might 
use env vars and use them almost off the shelf. 

-r

> On Jun 25, 2019, at 6:07 PM, Tyson Norris  wrote:
> 
> I had to read this several times, but have some suggestions. I think when you 
> say "action's arguments", you mean action-configured params, e.g. `wsk action 
> create --param p1 v1`?
> 
> My preferences would be:
> - we should split off "run" args into context and params - this is the 
> convention change for redefining main(args) as main(context, args) we have 
> discussed in the past. 
> - I support either having init receive action-configured params 
> - activation args that are possibly overridden should behave exactly as 
> specified args - is it important that action-configured args are actually 
> overridden, if the context and params are separated? (receive both values, 
> and logic must decide when to use which)
> - let's not use env variables for any arg that is variable per activation - 
> it is impossible if you support concurrency, and unneeded if we pass the 
> context to "run". 
> 
> Regarding Matt's suggestion to remove init - I like this idea, but I have 
> concerns compared to knative which might serve every function with a 
> different container, vs having some containers reused for multiple functions. 
> In the case where we init code into an already running container, it is 
> useful to have the init process separate from run, since otherwise each 
> runtime will need to track its own init state and queue requests during init 
> etc. If I'm not getting the whole picture with knative, please correct me.
> 
> 
> Thanks
> Tyson 
> 
> On 6/24/19, 8:43 AM, "Rodric Rabbah"  wrote:
> 
>In the current activation model, an action's arguments are always provided
>to the action on "run", not "init".
> 
>Should we consider partitioning the argument list into two sets, the first
>is exported as environment variables at "init" time, and the second become
>the action's argument at "run" time? A criteria for partitioning is that
>the environment variable starts with a capital letter, which is a common
>convention.
> 
>For example, an action which is invoked with a JSON object
> 
>{ "XYZ": true,
>  "abc" : false }
> 
>would receive {"abc": false} as its arguments and can read XYZ from the
>environment (as process.env.XYZ == "true" in Node.js).
> 
>This change would:
>1. require a change in the invoker to pass arguments during initialization
> 
>2. require a change in the runtime proxies to export the arguments to the
>environment at initialization time (additional work may be implied by 1b)
> 
>3. an annotation on actions to opt into this partitioning for backward
>compatibility or to opt out. For example '

Re: Openwhisk in a standalone runnable jar (#4516)

[Rodric Rabbah]

I tried this out - it is awesome... and fast!

-r

On Mon, Jun 24, 2019 at 6:08 AM Chetan Mehrotra 
wrote:

> Hi Team,
>
> At this stage the OpenWhisk Standalone jar works fine. So would like
> to get it tried out in real world and for that need some volunteers!
>
> You can follow the steps at
> https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
> and download the jar and try out a hello world or better some complex
> scenarios.
>
> In brief following steps would be needed to run this locally on your
> systems.
>
> $ wget
> https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar
> $ java -jar openwhisk-standalone.jar
> $ wsk property set --apihost 'http://localhost:3233' --auth
>
> '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP'
>
> Let me know your experience. If you face any issue or have some
> feedback then please share that on this thread or at [1]. Also check
> the Known Issues section at [2]
>
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/pull/4516
> [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
>
> On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra
>  wrote:
> >
> > Hi Nick,
> >
> > > we have the need to create namespaces/authkeys
> >
> > I have added an initial readme with steps on how to register custom
> > namespace at [1]. Can you give it a try and let us know if it meet
> > your requirements. Otherwise we can look into support a minimal REST
> > api to add user in standalone case
> >
> > Chetan Mehrotra
> > [1]
> https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces
> >
> > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell 
> wrote:
> > >
> > > we have the need to create namespaces/authkeys (so that we can run
> tests in
> > > parallel); is this supported by the current build env? or would we
> have to
> > > launch a separate openwhisk process for each tenant?
> > >
> > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana 
> wrote:
> > >
> > > > Chetan thanks for thinking on Windows users +1
> > > >
> > > > Just curious why the jar can’t be build in Windows can you log an
> issue
> > > > for this?
> > > >
> > > > My 2 cents for Windows or Mac I think the best way to build the jar
> would
> > > > be inside a docker container. A one liner with a single command
> should be
> > > > able to produce the jar on a local workstation or devops pipeline.
> > > >
> > > >
> > > > - Carlos Santana
> > > > @csantanapr
> > > >
> > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra <
> chetan.mehro...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Just to share some more progress. With runnable jar its now
> possible
> > > > > to run OpenWhisk standalone even on Windows and execute basic
> actions.
> > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10
> > > > >
> > > > > Note you would probably not be able to build the project on
> windows.
> > > > > But copying the produced jar would let you run it on Windows.
> > > > >
> > > > > Chetan Mehrotra
> > > > >
> > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra
> > > > >  wrote:
> > > > >>
> > > > >> Hi Nick,
> > > > >>
> > > > >> For now just building the PR branch and running following command
> > > > >> should get you going
> > > > >>
> > > > >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar
> > > > >>
> > > > >> You can use the default guest and whisk.system credentials to
> interact
> > > > >> with it. I am in the process of writing a readme for various
> options
> > > > >> exposed. Hope to get it done by Monday
> > > > >>
> > > > >> Chetan Mehrotra
> > > > >>
> > > > >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell <
> moose...@gmail.com>
> > > > wrote:
> > > > >>>
> > > > >>> thanks chetan for doing this!
> > > > >>>
> > > > >>> could you provide some example startup sequences, e.g. with
> sample
> > > > configs?
> > > > >>> i'm willing to try this out for our CI/CD pipeline -- i'm sick
> of 1)
> > > > >>> waiting 5-7 minutes for openwhisk to start up; and b) fighting
> ansible
> > > > >>> versus xenial :)
> > > > >>>
> > > > >>> @starpit
> > > > >>>
> > > > >>> On Fri, Jun 21, 2019 at 12:44 AM Chetan Mehrotra <
> > > > chetan.mehro...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > > What's the performance like on startup time
> > > > 
> > > >  It starts in < 5 secs.
> > > > 
> > > >  So far no major blocking issue apart from fetching docker logs
> on Mac.
> > > >  Current approach of directly reading the log json does not
> work. So
> > > >  need to have a mac version which uses `docker logs` command to
> > > >  somewhat handle such a case.
> > > > 
> > > >  Chetan Mehrotra
> > > > 
> > > > > On Thu, Jun 20, 2019 at 9:50 PM James Thomas <
> jthomas...@gmail.com>
> > > > wrote:
> > > > >
> > > > > That is mind-blowingly cool!
> > > > >
> > > > > 

exporting activation arguments to the environment

[Rodric Rabbah]

In the current activation model, an action's arguments are always provided
to the action on "run", not "init".

Should we consider partitioning the argument list into two sets, the first
is exported as environment variables at "init" time, and the second become
the action's argument at "run" time? A criteria for partitioning is that
the environment variable starts with a capital letter, which is a common
convention.

For example, an action which is invoked with a JSON object

{ "XYZ": true,
  "abc" : false }

would receive {"abc": false} as its arguments and can read XYZ from the
environment (as process.env.XYZ == "true" in Node.js).

This change would:
1. require a change in the invoker to pass arguments during initialization

2. require a change in the runtime proxies to export the arguments to the
environment at initialization time (additional work may be implied by 1b)

3. an annotation on actions to opt into this partitioning for backward
compatibility or to opt out. For example '-a env-partition-arguments true'
partitions the arguments and actions without this annotation are not
affected.

Some obvious question:
Q1a. should the invoker perform the partitioning or delegate it to the
runtime? The advantage of the former is that the runtimes do not have to
implement the filtering policy and do less work. I think it makes sense to
do this invoker side for uniformity.

Q1b. should the partitioning treat environment variables as immutable post
init and ignore the partition on warm starts? This is an issue when a value
is overridden during POST invoke only since for a webaction, you cannot
override a value that's already defined (and updating a bound parameter on
an action invalidates warm containers). I think env vars should be treated
as immutable despite the issue with POST invoke.

-r

Re: Re: Backpressure for slow activation storage in Invoker

[Rodric Rabbah]

> Can we handle these in same way as user events? Maybe exactly like user
events, as in use a single service to process both topics.

good call - the user events already contains much of the activation record
(if not all modulo the logs)?

-r

oauth token verification in the controller

[Rodric Rabbah]

I'm curious if anyone has thought about or implemented an oauth based
authentication mechanism in the controller. I've thought about replacing
the subject authentication with oauth and think it would not be a lot of
work to do although it does have some wider implications.

-r

Re: Backpressure for slow activation storage in Invoker

[Rodric Rabbah]

Overflowing to Kafka (option b) is better. Actually I would dump all the 
activations there and have a separate process to drain that Kafka topic to the 
datastore or logstore. 

There is another approach of routing the logs directly to a logstore without 
going through the invoker at all. IBM may have experimented with this maybe 
someone else can comment on that. 

-r

> On Jun 20, 2019, at 2:20 AM, Chetan Mehrotra  
> wrote:
> 
> Hi Team,
> 
> When rate of activation is high (specially with concurrency enabled)
> in a specific invoker then its possible that rate of storage of
> activation in ArtifactStore lags behind rate of activation record
> generation.
> 
> For CouchDB this was somewhat mitigated by using a Batcher
> implementation which internally used CouchDB bulk insert api
> (#2812)[1]. However currently Batcher is configured with a queue size
> of Int.max [2] which can potentially lead to Invoker going OOM
> 
> We tried to implement a similar support for CosmosDB (#4513)[3]. With
> our test we see that even with queue size of 100k was getting filled
> up quickly with higher load.
> 
> For #2812 Rodric had mentioned the need to support backpressure [4]
> 
>> we should perhaps open an issue to refactor the relevant code so that we can 
>> backpressure the invoker feed when the activations can't be drained fast 
>> enough.
> 
> Currently the storeActivation call is not waited upon in
> ContainerProxy and hence there is no backpressure. Wanted to check on
> what possible options we can try if activations can't be drained fast
> enough.
> 
> Option A - Wait till enqueue
> -
> 
> Somehow when calling storeACtivation wait till calls gets "enqueued".
> If it gets rejected due to queue full (assuming here that
> ArtifactStore has a queued storage) then we wait and retry few times.
> If it gets queued then we simply complete the call.
> 
> With this we would not be occupying the invoker slot untill storage is
> complete. Instead we just occupy bit more till activations get
> enqueued to in memory buffer
> 
> Option B - Overflow to Kafka and new OverflownActivationRecorderService
> 
> 
> With enqueuing the activations there is always a chance of increase
> the heap pressure specially if activation size is large (user
> controlled aspect). So another option would be to overflow to Kafka
> for storing activation.
> 
> If internal queue is full (queue size can now be small) then we would
> enque the record to Kafka. Kafka would in general have higher
> throughput rate compared to normal storage.
> 
> Then on other end we can have a new micro service which would poll
> this "overflowActivations" topic and then persist them in
> ArtifactStore. Here we can even use single but partitioned topic if
> need to scale out the queue processing by multiple service instances
> if needed.
> 
> Any feedback on possible option to pursue would be helpful!
> 
> Chetan Mehrotra
> [1]: https://github.com/apache/incubator-openwhisk/pull/2812
> [2]: 
> https://github.com/apache/incubator-openwhisk/blob/master/common/scala/src/main/scala/org/apache/openwhisk/core/database/Batcher.scala#L56
> [3]: https://github.com/apache/incubator-openwhisk/pull/4513
> [4]: 
> https://github.com/apache/incubator-openwhisk/pull/2812#pullrequestreview-67378126

Re: openwhisk distributions via dockerhub

[Rodric Rabbah]

>> 3. Once we have addressed (1) and (2),  we should consider opening a legal
>> discuss thread to see if we can continue to use /u/openwhisk (with clear
>> branding that /u/openwhisk is an official distribution channel from the
>> Apache OpenWhisk (P)PMC) or if we must migrate to /u/apacheopenwhisk or
>> similar.

Should we do that (ask legal) in parallel with 1/2?

-r

Re: openwhisk distributions via dockerhub

[Rodric Rabbah]

If we’re going to fix up all the builds we might as well use Apache/ on 
dockerhub or whatever we decide the location should be so we’re not doing this 
chore twice. 

I don’t know how we get access to the Apache org and if that’s tedious to 
manage vs an org we already own and manage. 

-r

> On Jun 19, 2019, at 3:43 PM, David P Grove  wrote:
> 
> 
> 
> Hi,
> 
> Another spinoff from the graduation discussion on the incubator general
> list relates to our project's use of dockerhub.  We were pointed to a set
> of (unofficial) distribution guidelines [1] that seem fairly sensible to
> me.  I've inlined the docker portion of [1]  at the end of this email.
> 
> Summarizing, I think there are at least 2 (and perhaps 3) action items for
> us to consider and implement.
> 
> 1. One set of actions is a per-image standardization of the dockerhub
> metadata for that image (overview, Dockerfile, disclaimer, etc).  Many of
> our images lack this consistent metadata.  A little tedious, but not that
> big of a deal.
> 
> 2. The big item is that I believe we need to re-engineer our CI/CD process
> across all of our git repos to replace the use of the 'latest' tag with
> 'nightly' (or similar).  This is going to be tedious and labor intensive
> for us, but given the special treatment of 'latest' by docker pull I
> believe it to be unavoidable.  It is a sound principle that a tagless
> 'docker pull openwhisk/' by a user should get an artifact that
> corresponds to an official release.  I believe the best implementation for
> this is having 'latest' be an alias for the most current official release,
> not as an alias for the latest nightly build.
> 
> 3. Once we have addressed (1) and (2),  we should consider opening a legal
> discuss thread to see if we can continue to use /u/openwhisk (with clear
> branding that /u/openwhisk is an official distribution channel from the
> Apache OpenWhisk (P)PMC) or if we must migrate to /u/apacheopenwhisk or
> similar.
> 
> --dave
> 
> [1]
> https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines
> 
> Docker
> 
> 
> Artifacts need to be placed in https://hub.docker.com/r/apache/ or
> https://hub.docker.com/u/apache/
> 
> 
> To comply with ASF release and distributions please ensure the following:
>  The overview should include the incubator disclaimer.
>  The docker file (if it exists) should include an ASF header.
>  The docker file (if it exists) should include the incubator
>  disclaimer.
>  docker pull apache/ should not install an artifact
>  containing unapproved code.
>  Release candidates, nightlies or snapshots need to be clearly tagged.
>  The latest tag should not point to an artifact containing unapproved
>  code e.g. to master or dev branches or to a RC or snapshot.
> 

Re: [ACTION REQUIRED] Nominations for PMC Chair for TLP graduation resolution

[Rodric Rabbah]

A one year rotation makes sense to me. 

I’d like to nominate Dave Grove for consideration. Dave is engaged on the 
general list, and I’ve consulted him many times on the Apache Way and for 
pointers to Apache docs. He’s also lead several releases and followed up on 
issues noted during voting to tighten our processes in line with Apache 
requirements. And he’s authored/submitted more than one report for our poddling 
(something I haven’t done yet ;).

-r

> On Jun 19, 2019, at 6:14 AM, Bertrand Delacretaz  
> wrote:
> 
>> On Wed, Jun 19, 2019 at 11:48 AM Justin Mclean  wrote:
>> ...Just a reminder, the chair is not a leadership role, but it it more a 
>> secretarial and admin role...
> 
> I was going to say that!
> 
> In extreme cases the chair might have to make decisions on behalf of
> the PMC but in 19 years at Apache I don't remember this happening.
> 
> The chair's role is really one of liaison with the Board, as Justin says.
> 
> That doesn't mean it's not a cool role, but we shouldn't put a heavier
> burden on their shoulders than this!
> 
> I support Rodric's nomination, and if there's a need note that we can
> also discuss and vote on this on the private list if preferred (as
> it's about people), and even use http://steve.apache.org/ for
> anonymous voting if people think this is required. I don't, but wanted
> to mention those options.
> 
> Also, I think it's good to rotate the chair's role regularly, so that
> more people know how that works. So I'd suggest agreeing that this
> nomination is for one year initially.
> 
> -Bertrand

Re: [ACTION REQUIRED] Nominations for PMC Chair for TLP graduation resolution

[Rodric Rabbah]

Thanks Dominic for the kind words. I'm humbled.

It's been wonderful seeing several committers emerge to also lead this
project and contribute in various ways. It would be amazing to see  others
also considered as a testament to the maturity of the project.

-r

On Tue, Jun 18, 2019 at 9:47 PM Dominic Kim  wrote:

> Aside from the duties of the Char, IMHO, a PMC Chair should be the one who
> loves the project and makes devotion to the project the most.
> Also, he should keep eyes on most of the works around the project and try
> to get more people involved in the project.
> He should lead the project in the right direction at the same time, he
> should always be nice and not arrogant.
>
> And I know who's exactly right for the position.
>
> I want to recommend Rodric Rabbah.
>
> If someone should be the Char, I think he is the one.
>
> Best regards
> Dominic
>
>
>
> 2019년 6월 19일 (수) 오전 5:33, Matt Sicker 님이 작성:
>
> > I'll note as a PMC Chair of another project here, it's not a heavy
> > burden at all, especially if you're already reading the mailing lists
> > and participating in release votes. Please be encouraged to volunteer
> > to chair the PMC!
> >
> > On Tue, 18 Jun 2019 at 12:15, Rodric Rabbah  wrote:
> > >
> > > This link http://www.apache.org/dev/pmc.html#chair is helpful to
> > understand
> > > the duties of the PMC chair.
> > >
> > > -r
> >
> >
> >
> > --
> > Matt Sicker 
> >
>

Re: status of openwhisk components releases

[Rodric Rabbah]

Vincent and Matt R. (apologies if I missed others) have done a great job
documenting the release steps via both the automated tooling and manual
process. The docs are here
https://github.com/apache/incubator-openwhisk-release/ and this is what I
followed when I release-managed the runtimes.

-r

On Sun, Jun 16, 2019 at 12:56 PM Matt Sicker  wrote:

> How well documented is release management here? Or how well automated would
> be an interesting question too.
>
> On Fri, Jun 14, 2019 at 14:27, Rodric Rabbah  wrote:
>
> > Per this issue
> > https://github.com/apache/incubator-openwhisk-release/issues/241 we are
> > almost ready to cut a new release of the openwhisk repo. The last release
> > of the main repo was 0.9 and I wonder if we should cut the new release at
> > 1.0. Thoughts?
> >
> > We need to create a release candidate for the catalog and then we're
> ready
> > to work on the main repo!
> >
> > Anyone interested in being a release manager for either repo? I'm told
> > there are swag rewards too.
> >
> > -r
> >
> --
> Matt Sicker 
>

Re: [ACTION REQUIRED] Nominations for PMC Chair for TLP graduation resolution

[Rodric Rabbah]

This link http://www.apache.org/dev/pmc.html#chair is helpful to understand
the duties of the PMC chair.

-r

Re: Accept NodeJs syntax quirks

[Rodric Rabbah]

Hey Martin - the refactoring was not intended to be semantic changing.
Thanks for reporting the bug. I prefer to not accept #4514 and instead
address this by restoring the behavior as noted in #136. I did add a test
in the latter inspired by the code that broke in the former.

-r

On Tue, Jun 18, 2019 at 11:49 AM Martin Henke  wrote:

> Rodrics latest updates to the nodeJs runtime uncovered that some of our
> NodeJs test actions are using variables in the global scope (by omitting
> the let or var keywords). The related tests are now failing in the Travis
> build.
>
> I opened https://github.com/apache/incubator-openwhisk/pull/4514 to fix
> the syntax of those actions.
> While this PR would fix the build problem, there would be the danger of
> breaking existing customer actions with the runtime changes.
>
> Therefore Rodric additionally opened a PR in the NodeJs runtime to allow
> these syntax quirks again.
> https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/136
>
> I do suggest to fix the problem by accepting his runtime PR and only
> optionally merge my core PR with the syntax changes
> after we see the problem go away.
>
> Regards,
> Martin

Re: Openwhisk in a standalone runnable jar (#4516)

[Rodric Rabbah]

That's very cool! +1.

-r

On Tue, Jun 18, 2019 at 9:22 AM Chetan Mehrotra 
wrote:

> Hi Team,
>
> Recently based on some feedback by end users I felt a need for a
> simpler way to test out OpenWhisk for end users. Towards that end
> there is a new PR #4516 which enables launching OpenWhisk as a simple
> runnable jar.
>
> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar
>
> Post this OpenWhisk server would be accessible over 8080 port and wsk
> can be configured to us that
>
> More details are provided at #4516 [1].
>
> Thoughts?
>
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/pull/4516
>

Re: [DISCUSS] - prepare to release OpenWhisk catalog

[Rodric Rabbah]

If the weather package is functional and can run with newer node kinds then I’d 
favor keeping it.

I’d favor removing the combinators as we have composer now and we didn’t 
document them properly (my fault).

I would consider removing the bash installers since they’re redundant with the 
wsk deploy manifest.

I’m indifferent on version number. 

-r

> On Jun 17, 2019, at 8:57 AM, David P Grove  wrote:
> 
> 
> 
> As Rodric mentioned in [1], the catalog is the last package we need to
> release before we can start on the main repo.
> 
> This is a discussion thread to figure out what (if anything) needs to be
> done before we can start the release process. Here are three questions I
> have:
> 
> 1. Should we remove the Weather Company package and let interested users
> get it from a downstream IBM repo, or is this a generally useful package
> that we should provide in the Apache release (like Slack and Github)?
> 
> 2. Should we remove the combinator package? (It is marked as deprecated in
> the package install scripts).  I think the functionality is now better
> provided by Apache OpenWhisk composer, but I don't know the full history of
> this package.
> 
> 3. Should this release be numbered 0.10.0 or 1.0.0?
> 
> --dave
> 
> [1]
> https://lists.apache.org/thread.html/ddf47c05e72426a47f0a414e1dd68cc6075fa4a60be1753318ebfe58@%3Cdev.openwhisk.apache.org%3E

status of openwhisk components releases

[Rodric Rabbah]

Per this issue
https://github.com/apache/incubator-openwhisk-release/issues/241 we are
almost ready to cut a new release of the openwhisk repo. The last release
of the main repo was 0.9 and I wonder if we should cut the new release at
1.0. Thoughts?

We need to create a release candidate for the catalog and then we're ready
to work on the main repo!

Anyone interested in being a release manager for either repo? I'm told
there are swag rewards too.

-r

refactoring nodejs runtime

[Rodric Rabbah]

I opened a PR to refactor and apply javascript-isms to our node.js runtime
https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/133.

There are no semantic changes intended only code hygiene applied.

-r

Re: managing our project trademarks

[Rodric Rabbah]

> For example, the official project plugins follow a naming scheme of
maven-foo-plugin, while third party plugins follow a naming scheme of
foo-maven-plugin. Similar to the distinction between “Apache Maven Foo
Plugin” versus “Foo Plugin for Apache Maven”.

This is useful - are you suggesting we document this on the trademarks page
also?


-r

managing our project trademarks

[Rodric Rabbah]

The discussion about our graduation on the general list [1] led to a
question relating to how we manage the project trademarks. Specifically,
our project website does not provide any guidance we can point to when
needed. We have previously discussed the use of the OpenWhisk name in a
prior instance [2] but did not document our guidance on the website. Apache
Spark is one example that might be useful for reference, available at
http://spark.apache.org/trademarks.html. It provides appropriate links to
the ASF [3] and summarizes the key points of the policy.

This email is to solicit help authoring our own project trademarks policy
summary page, and to solicit feedback on additional points to summarize
relating to the use of the name OpenWhisk.

For example, how do we handle the use of OpenWhisk in a GitHub repository
name? A small sample of searches on popular Apache project names leads one
to believe this is quite common.  In some cases it may be sufficient for
the repository to use the full project name "Apache OpenWhisk" prominently
in the project description and first mentions in the documentation (with
links to the project website). When do we require an acknowledgement of the
trademark, or ask a repository to be renamed?

-r

[1]
https://lists.apache.org/thread.html/02dd71dcc6b2f326015954388c61d06d12f982fd64f34ee40e076171@%3Cgeneral.incubator.apache.org%3E

[2]
https://lists.apache.org/thread.html/99e2431472d015fb23af38d1c2dab47ef9eb5a8a980cf2b80ca0035b@%3Cdev.openwhisk.apache.org%3E

[3] https://www.apache.org/foundation/marks/faq/#products

Re: our source releases do not include tests

[Rodric Rabbah]

I’m not sure why we don’t include tests. I do remember this mentioned in the 
past I’ll see if I can find the discussion on the dev list.

-r

> On Jun 10, 2019, at 6:08 PM, David P Grove  wrote:
> 
> 
> 
> I was trying to understand Justin's comment [1] that the NOTICE file for
> apigateway referred to MIT licensed files that weren't in the release.
> 
> I eventually realized this is because the script that makes the release
> excludes any top-level directory called 'tests' (among other things that
> seem plausible to exclude).  [2]  This seems a bit odd to me.  Do we really
> want to do this?
> 
> --dave
> 
> [1]
> https://lists.apache.org/thread.html/1bb15cc18ba446c8102e1ec8780cf0e05fcab345bf73ed3caae4b4bd@%3Cgeneral.incubator.apache.org%3E
> [2]
> https://github.com/apache/incubator-openwhisk-release/blob/38e5c15a9f3405f30ad88b5594618aa27d7b9019/tools/package_source_code.sh#L33

[VOTE] [RESULT] Apache OpenWhisk graduation to Top Level Project

[Rodric Rabbah]

Voting is now closed. Thanks to everyone who has participated.

The vote has passed.

We received 19 votes for "+1 Apache OpenWhisk should graduate." and no
other votes.

Voting +1 were all three project mentors (Matt Sicker, Bertrand
Delacretaz, Krzysztof Sobkowiak) and David Breitgand, Tyson Norris,
Dominic Kim, Matt Hamann, James Thomas, Carlos Santana, Dascalita
Dragos, Chetan Mehrotra, Dave Grove, Olivier Tardieu,  Priti Desai,
Michele Sciabarra, Rob Allen, Erez Hadad, Matt Rutkowski, Vincent S
Hou,

The permalink for the voting thread is [1].

-r

[1] 
https://lists.apache.org/thread.html/2987aa4ebd884fdb615f5c178399db6c863b000560fce8162055a23b@%3Cdev.openwhisk.apache.org%3E