Re: Openwhisk in a standalone runnable jar (#4516)

[Chetan Mehrotra]

> What's the performance like on startup time

It starts in < 5 secs.

So far no major blocking issue apart from fetching docker logs on Mac.
Current approach of directly reading the log json does not work. So
need to have a mac version which uses `docker logs` command to
somewhat handle such a case.

Chetan Mehrotra

On Thu, Jun 20, 2019 at 9:50 PM James Thomas  wrote:
>
> That is mind-blowingly cool!
>
> What's the performance like on startup time?
>
> On Thu, 20 Jun 2019 at 06:14, Carlos Santana  wrote:
> >
> > Genius!
> > https://www.adminsub.net/tcp-udp-port-finder/whisker
> >
> >
> > - Carlos Santana
> > @csantanapr
> >
> > > On Jun 19, 2019, at 12:30 PM, David P Grove  wrote:
> > >
> > > WhiskerControl
>
>
>
> --
> Regards,
> James Thomas

Re: Backpressure for slow activation storage in Invoker

[Rodric Rabbah]

Overflowing to Kafka (option b) is better. Actually I would dump all the 
activations there and have a separate process to drain that Kafka topic to the 
datastore or logstore. 

There is another approach of routing the logs directly to a logstore without 
going through the invoker at all. IBM may have experimented with this maybe 
someone else can comment on that. 

-r

> On Jun 20, 2019, at 2:20 AM, Chetan Mehrotra  
> wrote:
> 
> Hi Team,
> 
> When rate of activation is high (specially with concurrency enabled)
> in a specific invoker then its possible that rate of storage of
> activation in ArtifactStore lags behind rate of activation record
> generation.
> 
> For CouchDB this was somewhat mitigated by using a Batcher
> implementation which internally used CouchDB bulk insert api
> (#2812)[1]. However currently Batcher is configured with a queue size
> of Int.max [2] which can potentially lead to Invoker going OOM
> 
> We tried to implement a similar support for CosmosDB (#4513)[3]. With
> our test we see that even with queue size of 100k was getting filled
> up quickly with higher load.
> 
> For #2812 Rodric had mentioned the need to support backpressure [4]
> 
>> we should perhaps open an issue to refactor the relevant code so that we can 
>> backpressure the invoker feed when the activations can't be drained fast 
>> enough.
> 
> Currently the storeActivation call is not waited upon in
> ContainerProxy and hence there is no backpressure. Wanted to check on
> what possible options we can try if activations can't be drained fast
> enough.
> 
> Option A - Wait till enqueue
> -
> 
> Somehow when calling storeACtivation wait till calls gets "enqueued".
> If it gets rejected due to queue full (assuming here that
> ArtifactStore has a queued storage) then we wait and retry few times.
> If it gets queued then we simply complete the call.
> 
> With this we would not be occupying the invoker slot untill storage is
> complete. Instead we just occupy bit more till activations get
> enqueued to in memory buffer
> 
> Option B - Overflow to Kafka and new OverflownActivationRecorderService
> 
> 
> With enqueuing the activations there is always a chance of increase
> the heap pressure specially if activation size is large (user
> controlled aspect). So another option would be to overflow to Kafka
> for storing activation.
> 
> If internal queue is full (queue size can now be small) then we would
> enque the record to Kafka. Kafka would in general have higher
> throughput rate compared to normal storage.
> 
> Then on other end we can have a new micro service which would poll
> this "overflowActivations" topic and then persist them in
> ArtifactStore. Here we can even use single but partitioned topic if
> need to scale out the queue processing by multiple service instances
> if needed.
> 
> Any feedback on possible option to pursue would be helpful!
> 
> Chetan Mehrotra
> [1]: https://github.com/apache/incubator-openwhisk/pull/2812
> [2]: 
> https://github.com/apache/incubator-openwhisk/blob/master/common/scala/src/main/scala/org/apache/openwhisk/core/database/Batcher.scala#L56
> [3]: https://github.com/apache/incubator-openwhisk/pull/4513
> [4]: 
> https://github.com/apache/incubator-openwhisk/pull/2812#pullrequestreview-67378126

Re: openwhisk distributions via dockerhub

[Rodric Rabbah]

>> 3. Once we have addressed (1) and (2),  we should consider opening a legal
>> discuss thread to see if we can continue to use /u/openwhisk (with clear
>> branding that /u/openwhisk is an official distribution channel from the
>> Apache OpenWhisk (P)PMC) or if we must migrate to /u/apacheopenwhisk or
>> similar.

Should we do that (ask legal) in parallel with 1/2?

-r

Re: [DISCUSS] - prepare to release OpenWhisk catalog

[James Thomas]

+1 to everything Rodric said!

On Mon, 17 Jun 2019 at 14:07, Rodric Rabbah  wrote:
>
> If the weather package is functional and can run with newer node kinds then 
> I’d favor keeping it.
>
> I’d favor removing the combinators as we have composer now and we didn’t 
> document them properly (my fault).
>
> I would consider removing the bash installers since they’re redundant with 
> the wsk deploy manifest.
>
> I’m indifferent on version number.
>
> -r
>
> > On Jun 17, 2019, at 8:57 AM, David P Grove  wrote:
> >
> >
> >
> > As Rodric mentioned in [1], the catalog is the last package we need to
> > release before we can start on the main repo.
> >
> > This is a discussion thread to figure out what (if anything) needs to be
> > done before we can start the release process. Here are three questions I
> > have:
> >
> > 1. Should we remove the Weather Company package and let interested users
> > get it from a downstream IBM repo, or is this a generally useful package
> > that we should provide in the Apache release (like Slack and Github)?
> >
> > 2. Should we remove the combinator package? (It is marked as deprecated in
> > the package install scripts).  I think the functionality is now better
> > provided by Apache OpenWhisk composer, but I don't know the full history of
> > this package.
> >
> > 3. Should this release be numbered 0.10.0 or 1.0.0?
> >
> > --dave
> >
> > [1]
> > https://lists.apache.org/thread.html/ddf47c05e72426a47f0a414e1dd68cc6075fa4a60be1753318ebfe58@%3Cdev.openwhisk.apache.org%3E



-- 
Regards,
James Thomas

Re: openwhisk distributions via dockerhub

[James Thomas]

On Wed, 19 Jun 2019 at 20:44, David P Grove  wrote:
>
> 3. Once we have addressed (1) and (2),  we should consider opening a legal
> discuss thread to see if we can continue to use /u/openwhisk (with clear
> branding that /u/openwhisk is an official distribution channel from the
> Apache OpenWhisk (P)PMC) or if we must migrate to /u/apacheopenwhisk or
> similar.

If that discussion leads us to believe `openwhisk` must be retired in
favour the full project name, this will also affect the NPM module for
the JS Client SDK.
https://www.npmjs.com/package/openwhisk

Looking over the NPM guidelines from ASF, we comply with all of them
apart from the name. I've opened an issue to track this and will wait
on the legal discussion.
https://github.com/apache/incubator-openwhisk-client-js/issues/169

-- 
Regards,
James Thomas

Re: Re: [ACTION REQUIRED] Nominations for PMC Chair for TLP graduation resolution

[James Thomas]

+1 on either Dave or Rodric taking the chair. Both are excellent
candidates and ideally suited to the role.

On Wed, 19 Jun 2019 at 20:56, David P Grove  wrote:
>
> Rodric Rabbah  wrote on 06/19/2019 06:39:19 AM:
> >
> > A one year rotation makes sense to me.
>
> Big +1 on planning for rotation.
>
> > I’d like to nominate Dave Grove for consideration.
>
> Thanks Rodric.  I'm happy to take a turn at some point.  Whether that's
> soon or a couple years down the road doesn't matter to me.
>
> -dave



-- 
Regards,
James Thomas

Re: Openwhisk in a standalone runnable jar (#4516)

[James Thomas]

That is mind-blowingly cool!

What's the performance like on startup time?

On Thu, 20 Jun 2019 at 06:14, Carlos Santana  wrote:
>
> Genius!
> https://www.adminsub.net/tcp-udp-port-finder/whisker
>
>
> - Carlos Santana
> @csantanapr
>
> > On Jun 19, 2019, at 12:30 PM, David P Grove  wrote:
> >
> > WhiskerControl



-- 
Regards,
James Thomas

[slack-digest] [2019-06-19] #composer

[OpenWhisk Team Slack]

2019-06-19 09:19:59 UTC - Bruno Girin: Thanks @Olivier Tardieu I'll do that 
then!
https://openwhisk-team.slack.com/archives/C7DJNS37W/p1560935999004900

[slack-digest] [2019-06-19] #general

[OpenWhisk Team Slack]

2019-06-19 19:00:28 UTC - rawkintrevo: hey- not sure if any of y'all are bikers 
but I did a thing then blogged about it- Apache OpenWhisk- keeping people safe 

heart : Rodric Rabbah
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560970828076500

2019-06-19 19:19:10 UTC - rawkintrevo: as an aside- has anyone ever seen a 
shoping cart w apache openwhisk demo?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560971950077000?thread_ts=1560971950.077000=C3TPCAQG1

2019-06-19 19:25:40 UTC - Rodric Rabbah: that’s really neat @rawkintrevo
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560972340077200

2019-06-19 19:25:53 UTC - Rodric Rabbah: is your blog also mentioned on twitter?
clap : Carlos Santana
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560972353077700

2019-06-19 19:26:03 UTC - rawkintrevo: it is- one sec, i'll give you the link
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560972363077900

2019-06-19 19:26:10 UTC - rawkintrevo: 

+1 : Rodric Rabbah, Rob Allen
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560972370078200

2019-06-19 19:27:41 UTC - Rodric Rabbah: i haven’t - i’ve done a stock trading 
app though
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1560972461078500?thread_ts=1560971950.077000=C3TPCAQG1

Backpressure for slow activation storage in Invoker

[Chetan Mehrotra]

Hi Team,

When rate of activation is high (specially with concurrency enabled)
in a specific invoker then its possible that rate of storage of
activation in ArtifactStore lags behind rate of activation record
generation.

For CouchDB this was somewhat mitigated by using a Batcher
implementation which internally used CouchDB bulk insert api
(#2812)[1]. However currently Batcher is configured with a queue size
of Int.max [2] which can potentially lead to Invoker going OOM

We tried to implement a similar support for CosmosDB (#4513)[3]. With
our test we see that even with queue size of 100k was getting filled
up quickly with higher load.

For #2812 Rodric had mentioned the need to support backpressure [4]

> we should perhaps open an issue to refactor the relevant code so that we can 
> backpressure the invoker feed when the activations can't be drained fast 
> enough.

Currently the storeActivation call is not waited upon in
ContainerProxy and hence there is no backpressure. Wanted to check on
what possible options we can try if activations can't be drained fast
enough.

Option A - Wait till enqueue
-

Somehow when calling storeACtivation wait till calls gets "enqueued".
If it gets rejected due to queue full (assuming here that
ArtifactStore has a queued storage) then we wait and retry few times.
If it gets queued then we simply complete the call.

With this we would not be occupying the invoker slot untill storage is
complete. Instead we just occupy bit more till activations get
enqueued to in memory buffer

Option B - Overflow to Kafka and new OverflownActivationRecorderService


With enqueuing the activations there is always a chance of increase
the heap pressure specially if activation size is large (user
controlled aspect). So another option would be to overflow to Kafka
for storing activation.

If internal queue is full (queue size can now be small) then we would
enque the record to Kafka. Kafka would in general have higher
throughput rate compared to normal storage.

Then on other end we can have a new micro service which would poll
this "overflowActivations" topic and then persist them in
ArtifactStore. Here we can even use single but partitioned topic if
need to scale out the queue processing by multiple service instances
if needed.

Any feedback on possible option to pursue would be helpful!

Chetan Mehrotra
[1]: https://github.com/apache/incubator-openwhisk/pull/2812
[2]: 
https://github.com/apache/incubator-openwhisk/blob/master/common/scala/src/main/scala/org/apache/openwhisk/core/database/Batcher.scala#L56
[3]: https://github.com/apache/incubator-openwhisk/pull/4513
[4]: 
https://github.com/apache/incubator-openwhisk/pull/2812#pullrequestreview-67378126