docker pull rate limiting

[Nick Mitchell]

Have you applied to dockerhub to exempt the openwhisk org from their
new-ish rate limiting? Travis tests (or any tests on a shared
infrastructure) will begin to routinely fail because of this:
>
>  docker: toomanyrequests: You have reached your pull rate limit. You may 
> increase the limit by authenticating and upgrading: 
> https://www.docker.com/increase-rate-limit.
>
> This, for example, can happen when pulling the openwhisk/standalone image,
or any of the action/skeleton base images, as part of automated tests.

@starpit

Re: [wskcli] Delete all entities in a namespace

[Nick Mitchell]

i think the core problem is that there are several layers of collections,
but no bulk and no cursored operations against these collections: "all
actions", "actions in package"... retries and careful coding can work
around these. so it might be sufficient just do it "twice" (in the
openwhisk client npm, and in the go CLI). our code is open source. it's
still not 100% correct. we still get travis failures 1/100 times or so.

On Tue, Mar 24, 2020 at 9:09 PM Carlos Santana  wrote:

> Hi Nick this is very useful even outside demos, I remember having a bash
> alias that it was 95% correct to delete all.
>
> Your proposing the heavy lifting is implemented on the server side
> controller API and the client/api just do a single http request “delete
> all” ?
>
> - Carlos Santana
> @csantanapr
>
> > On Mar 24, 2020, at 7:16 PM, Nick Mitchell  wrote:
> >
> > we implemented this in https://github.com/kui-shell/oui
> >
> > some lessons learned. it's possible, but tricky, due to the loose
> semantics
> > of openwhisk delete-after-delete ordering, plus some bits of potential
> race
> > conditions with with list windowing. things may have improved since we
> > first implemented this, so keep that in mind:
> >
> > 1) if you have an action in a package; then you have to delete the
> actions
> > in the package before you can delete the package; but... if you issue the
> > package delete after the last action delete API call has returned, there
> > (at least used to be) a short window where the package delete would
> fail. i
> > think this depends upon the quorum configuration of your backend
> database.
> > so you may need to have retries here.
> >
> > 2) if you have more than 200 (or whatever the LIMIT config is of your
> > server) actions (c.f. packages, etc.), then you will need to manage the
> > windowing yourself. since the openwhisk API has no strong sense of
> cursors
> > (or does it now?), you have to be careful to fetch all windows before
> > initiating any deletes in any of the windows...
> >
> > there may be more issues, but this one was non-trivial to get right.
> after
> > doing all the work, "delete all" struck me as the kind of thing that
> should
> > be supported by the API directly.
> >
> > @starpit
> >
> >> On Tue, Mar 24, 2020 at 5:23 PM Will Plusnick 
> wrote:
> >>
> >> Hello all!
> >>
> >> I've frequently found myself having to clear a namespace after working
> >> on a demo or when experimenting. I'll have a number of artifacts left
> >> in a namespace that I don't want to preserve. Right now, unless I used
> >> wskdeploy for deployment, I'm stuck manually deleting all of the
> >> actions, rules, etc. I want to propose adding a command to the
> >> namespace subcommand in the cli for deleting all the artifacts in a
> >> namespace.
> >>
> >> I envision the syntax for the command going something like:
> >>
> >> $ wsk namespace delete all
> >> This command will delete all entities in the namespace :
> >> 
> >> Are you sure you want to proceed? Type '' to proceed:
> >> 
> >> action:
> >>  is deleted
> >>  is deleted
> >> ..
> >> trigger:
> >>  is deleted
> >>  is deleted
> >> ..
> >> etc
> >>
> >> The user would then be prompted whether they really want to delete all
> >> of the entities in the namespace. They would need to type in a "danger
> >> phrase" such as the name of the namespace to avoid accidentally running
> >> the command. This is similar to deleting a github repo. If anything
> >> but the "danger phrase" is received then the operation is aborted.
> >>
> >> This is a feature I would find personally useful, and would like to
> >> add it to the cli. Does the community at large feel this is a useful
> >> feature?
> >>
> >> Thanks,
> >> Will
> >>
> >>
>

Re: Openwhisk in a standalone runnable jar (#4516)

[Nick Mitchell]

oh, one thing worth mentioning: when testing on electron, on linux, i have
to use an http keep-alive http agent when talking to the standalone. this
has been a perennial problem with electron/linux versus openwhisk (i still
have no idea why); but the standalone jar made the problem considerably
worse. i don't know whether this is an openwhisk bug or a linux bug or an
electron bug, but something breaks if electron+linux makes too many
connections to openwhisk.

On Mon, Jun 24, 2019 at 8:31 AM Nick Mitchell  wrote:

> it's been working well for our CI/CD tests. standalone openwhisk, combined
> with microk8s, are a perfect combination for test pipelines against this
> kind of infrastructure.
>
> thanks!!
>
> On Mon, Jun 24, 2019 at 6:08 AM Chetan Mehrotra 
> wrote:
>
>> Hi Team,
>>
>> At this stage the OpenWhisk Standalone jar works fine. So would like
>> to get it tried out in real world and for that need some volunteers!
>>
>> You can follow the steps at
>> https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
>> and download the jar and try out a hello world or better some complex
>> scenarios.
>>
>> In brief following steps would be needed to run this locally on your
>> systems.
>>
>> $ wget
>> https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar
>> $ java -jar openwhisk-standalone.jar
>> $ wsk property set --apihost 'http://localhost:3233' --auth
>>
>> '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP'
>>
>> Let me know your experience. If you face any issue or have some
>> feedback then please share that on this thread or at [1]. Also check
>> the Known Issues section at [2]
>>
>> Chetan Mehrotra
>> [1] https://github.com/apache/incubator-openwhisk/pull/4516
>> [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
>>
>> On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra
>>  wrote:
>> >
>> > Hi Nick,
>> >
>> > > we have the need to create namespaces/authkeys
>> >
>> > I have added an initial readme with steps on how to register custom
>> > namespace at [1]. Can you give it a try and let us know if it meet
>> > your requirements. Otherwise we can look into support a minimal REST
>> > api to add user in standalone case
>> >
>> > Chetan Mehrotra
>> > [1]
>> https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces
>> >
>> > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell 
>> wrote:
>> > >
>> > > we have the need to create namespaces/authkeys (so that we can run
>> tests in
>> > > parallel); is this supported by the current build env? or would we
>> have to
>> > > launch a separate openwhisk process for each tenant?
>> > >
>> > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana 
>> wrote:
>> > >
>> > > > Chetan thanks for thinking on Windows users +1
>> > > >
>> > > > Just curious why the jar can’t be build in Windows can you log an
>> issue
>> > > > for this?
>> > > >
>> > > > My 2 cents for Windows or Mac I think the best way to build the jar
>> would
>> > > > be inside a docker container. A one liner with a single command
>> should be
>> > > > able to produce the jar on a local workstation or devops pipeline.
>> > > >
>> > > >
>> > > > - Carlos Santana
>> > > > @csantanapr
>> > > >
>> > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra <
>> chetan.mehro...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > Just to share some more progress. With runnable jar its now
>> possible
>> > > > > to run OpenWhisk standalone even on Windows and execute basic
>> actions.
>> > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10
>> > > > >
>> > > > > Note you would probably not be able to build the project on
>> windows.
>> > > > > But copying the produced jar would let you run it on Windows.
>> > > > >
>> > > > > Chetan Mehrotra
>> > > > >
>> > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra
>> > > > >  wrote:
>> > > > >>
>> > > > >> Hi Nick,
>

Re: Openwhisk in a standalone runnable jar (#4516)

[Nick Mitchell]

it's been working well for our CI/CD tests. standalone openwhisk, combined
with microk8s, are a perfect combination for test pipelines against this
kind of infrastructure.

thanks!!

On Mon, Jun 24, 2019 at 6:08 AM Chetan Mehrotra 
wrote:

> Hi Team,
>
> At this stage the OpenWhisk Standalone jar works fine. So would like
> to get it tried out in real world and for that need some volunteers!
>
> You can follow the steps at
> https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
> and download the jar and try out a hello world or better some complex
> scenarios.
>
> In brief following steps would be needed to run this locally on your
> systems.
>
> $ wget
> https://github.com/chetanmeh/incubator-openwhisk/releases/download/v0.10/openwhisk-standalone.jar
> $ java -jar openwhisk-standalone.jar
> $ wsk property set --apihost 'http://localhost:3233' --auth
>
> '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP'
>
> Let me know your experience. If you face any issue or have some
> feedback then please share that on this thread or at [1]. Also check
> the Known Issues section at [2]
>
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/pull/4516
> [2] https://github.com/chetanmeh/incubator-openwhisk/releases/tag/v0.10
>
> On Fri, Jun 21, 2019 at 8:00 PM Chetan Mehrotra
>  wrote:
> >
> > Hi Nick,
> >
> > > we have the need to create namespaces/authkeys
> >
> > I have added an initial readme with steps on how to register custom
> > namespace at [1]. Can you give it a try and let us know if it meet
> > your requirements. Otherwise we can look into support a minimal REST
> > api to add user in standalone case
> >
> > Chetan Mehrotra
> > [1]
> https://github.com/chetanmeh/incubator-openwhisk/blob/openwhisk-standalone/core/standalone/README.md#adding-custom-namespaces
> >
> > On Fri, Jun 21, 2019 at 5:36 PM Nick Mitchell 
> wrote:
> > >
> > > we have the need to create namespaces/authkeys (so that we can run
> tests in
> > > parallel); is this supported by the current build env? or would we
> have to
> > > launch a separate openwhisk process for each tenant?
> > >
> > > On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana 
> wrote:
> > >
> > > > Chetan thanks for thinking on Windows users +1
> > > >
> > > > Just curious why the jar can’t be build in Windows can you log an
> issue
> > > > for this?
> > > >
> > > > My 2 cents for Windows or Mac I think the best way to build the jar
> would
> > > > be inside a docker container. A one liner with a single command
> should be
> > > > able to produce the jar on a local workstation or devops pipeline.
> > > >
> > > >
> > > > - Carlos Santana
> > > > @csantanapr
> > > >
> > > > > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra <
> chetan.mehro...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Just to share some more progress. With runnable jar its now
> possible
> > > > > to run OpenWhisk standalone even on Windows and execute basic
> actions.
> > > > > Tested it with Docker on Windows 2.0.0.3 on Windows 10
> > > > >
> > > > > Note you would probably not be able to build the project on
> windows.
> > > > > But copying the produced jar would let you run it on Windows.
> > > > >
> > > > > Chetan Mehrotra
> > > > >
> > > > > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra
> > > > >  wrote:
> > > > >>
> > > > >> Hi Nick,
> > > > >>
> > > > >> For now just building the PR branch and running following command
> > > > >> should get you going
> > > > >>
> > > > >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar
> > > > >>
> > > > >> You can use the default guest and whisk.system credentials to
> interact
> > > > >> with it. I am in the process of writing a readme for various
> options
> > > > >> exposed. Hope to get it done by Monday
> > > > >>
> > > > >> Chetan Mehrotra
> > > > >>
> > > > >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell <
> moose...@gmail.com>
> > > > wrote:
> > > > >>>
> > > > >>> thanks chetan for doing this!
> &g

Re: Openwhisk in a standalone runnable jar (#4516)

[Nick Mitchell]

we have the need to create namespaces/authkeys (so that we can run tests in
parallel); is this supported by the current build env? or would we have to
launch a separate openwhisk process for each tenant?

On Fri, Jun 21, 2019 at 8:03 AM Carlos Santana  wrote:

> Chetan thanks for thinking on Windows users +1
>
> Just curious why the jar can’t be build in Windows can you log an issue
> for this?
>
> My 2 cents for Windows or Mac I think the best way to build the jar would
> be inside a docker container. A one liner with a single command should be
> able to produce the jar on a local workstation or devops pipeline.
>
>
> - Carlos Santana
> @csantanapr
>
> > On Jun 21, 2019, at 7:45 AM, Chetan Mehrotra 
> wrote:
> >
> > Just to share some more progress. With runnable jar its now possible
> > to run OpenWhisk standalone even on Windows and execute basic actions.
> > Tested it with Docker on Windows 2.0.0.3 on Windows 10
> >
> > Note you would probably not be able to build the project on windows.
> > But copying the produced jar would let you run it on Windows.
> >
> > Chetan Mehrotra
> >
> > On Fri, Jun 21, 2019 at 5:13 PM Chetan Mehrotra
> >  wrote:
> >>
> >> Hi Nick,
> >>
> >> For now just building the PR branch and running following command
> >> should get you going
> >>
> >> java -jar openwhisk-standalone-1.0.0-SNAPSHOT.jar
> >>
> >> You can use the default guest and whisk.system credentials to interact
> >> with it. I am in the process of writing a readme for various options
> >> exposed. Hope to get it done by Monday
> >>
> >> Chetan Mehrotra
> >>
> >>> On Fri, Jun 21, 2019 at 4:57 PM Nick Mitchell 
> wrote:
> >>>
> >>> thanks chetan for doing this!
> >>>
> >>> could you provide some example startup sequences, e.g. with sample
> configs?
> >>> i'm willing to try this out for our CI/CD pipeline -- i'm sick of 1)
> >>> waiting 5-7 minutes for openwhisk to start up; and b) fighting ansible
> >>> versus xenial :)
> >>>
> >>> @starpit
> >>>
> >>> On Fri, Jun 21, 2019 at 12:44 AM Chetan Mehrotra <
> chetan.mehro...@gmail.com>
> >>> wrote:
> >>>
> >>>>> What's the performance like on startup time
> >>>>
> >>>> It starts in < 5 secs.
> >>>>
> >>>> So far no major blocking issue apart from fetching docker logs on Mac.
> >>>> Current approach of directly reading the log json does not work. So
> >>>> need to have a mac version which uses `docker logs` command to
> >>>> somewhat handle such a case.
> >>>>
> >>>> Chetan Mehrotra
> >>>>
> >>>>> On Thu, Jun 20, 2019 at 9:50 PM James Thomas 
> wrote:
> >>>>>
> >>>>> That is mind-blowingly cool!
> >>>>>
> >>>>> What's the performance like on startup time?
> >>>>>
> >>>>> On Thu, 20 Jun 2019 at 06:14, Carlos Santana 
> >>>> wrote:
> >>>>>>
> >>>>>> Genius!
> >>>>>> https://www.adminsub.net/tcp-udp-port-finder/whisker
> >>>>>>
> >>>>>>
> >>>>>> - Carlos Santana
> >>>>>> @csantanapr
> >>>>>>
> >>>>>>> On Jun 19, 2019, at 12:30 PM, David P Grove 
> >>>> wrote:
> >>>>>>>
> >>>>>>> WhiskerControl
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Regards,
> >>>>> James Thomas
> >>>>
>

Re: Openwhisk in a standalone runnable jar (#4516)

[Nick Mitchell]

thanks chetan for doing this!

could you provide some example startup sequences, e.g. with sample configs?
i'm willing to try this out for our CI/CD pipeline -- i'm sick of 1)
waiting 5-7 minutes for openwhisk to start up; and b) fighting ansible
versus xenial :)

@starpit

On Fri, Jun 21, 2019 at 12:44 AM Chetan Mehrotra 
wrote:

> > What's the performance like on startup time
>
> It starts in < 5 secs.
>
> So far no major blocking issue apart from fetching docker logs on Mac.
> Current approach of directly reading the log json does not work. So
> need to have a mac version which uses `docker logs` command to
> somewhat handle such a case.
>
> Chetan Mehrotra
>
> On Thu, Jun 20, 2019 at 9:50 PM James Thomas  wrote:
> >
> > That is mind-blowingly cool!
> >
> > What's the performance like on startup time?
> >
> > On Thu, 20 Jun 2019 at 06:14, Carlos Santana 
> wrote:
> > >
> > > Genius!
> > > https://www.adminsub.net/tcp-udp-port-finder/whisker
> > >
> > >
> > > - Carlos Santana
> > > @csantanapr
> > >
> > > > On Jun 19, 2019, at 12:30 PM, David P Grove 
> wrote:
> > > >
> > > > WhiskerControl
> >
> >
> >
> > --
> > Regards,
> > James Thomas
>

Re: Apache OpenWhisk Web Action HTTP Proxy

[Nick Mitchell]

this is great james, thanks for the awesome writeup. last year, i tried a
somewhat similar experiment with using openwhisk as a proxy to kubectl and
helm. it worked! but ended up not being viable due to cold starts. i either
needed to use a docker action to host the go binaries, or download them on
the fly. the latter, surprisingly, turned out to be faster... and the
results are borderline usable (1.2s cold start times, which was pretty nice
considering this included downloading a binary!).

but, compared to the 350ms typical response times from the base CLI, and
the 400ms or so response times to be had from a pod-hosted kubectl proxy...
i had to ditch the experiment before long.

thoughts here? i'd love to have used the serverless openwhisk-kubectl
proxy. did i miss something that would have made this a viable approach?

@starpit

On Tue, Apr 30, 2019 at 6:14 AM James Thomas  wrote:

> Hello Whiskers.
>
> I wanted to share an experiment I built recently to run existing web apps
> on openwhisk (with minimal changes) using custom Docker images support.
>
> https://github.com/jthomas/openwhisk-web-action-http-proxy
>
> It uses a HTTP proxy binary inside the runtime container to proxy incoming
> Web Action requests to the web apps run on a different port inside the same
> container. HTTP responses are returned to the platform as Web Action
> responses.
>
> For simple & stateless web apps - it worked better than I thought. Here's
> an example of how to use this with an exsting Node.js+Express Web App:
>
>
> https://github.com/jthomas/openwhisk-web-action-http-proxy#nodejs--express-example
>
> $ git clone https://github.com/jthomas/express_example
> $ cd express_example & npm install
> $ zip -r action.zip *
> $ wsk action create --docker jamesthomas/generic_node_proxy --web true
> --main "npm start" -p "__ow_proxy_port" 3000 web_app action.zip
> $ wsk action get web_app --url
>
> Two different options are available for getting web application source
> files into the runtime environment - runtime injection using zip files or
> putting source files directly into the runtime image.
>
> There's lots more details about the project in the project README and I've
> now written a blog post in more details about the project here:
>
> http://jamesthom.as/blog/2019/04/29/apache-openwhisk-web-action-http-proxy/
>
> --
> Regards,
> James Thomas
>

Re: Autonomous Container Scheduler v2 proposal

[Nick Mitchell]

>
> 1100+ Kafka brokers, over 31,000 topics


am i reading this right, as this seems like an unhealthy ratio. a new
broker for every 30 topics? so i must be misinterpreting the numbers...

On Wed, Nov 28, 2018 at 10:18 PM Dominic Kim  wrote:

> Dear Rodric.
>
> We`ve just started benchmarking of the scheduler with a variety of
> scenarios.
> I will come with more results.
>
> If you ask that because of scalability of Kafka, I have considered Kafka as
> a very scalable solution.
>
> Linkedin announced that they are utilizing a huge Kafka cluster.
>
> https://events.static.linuxfound.org/sites/events/files/slides/Kafka%20At%20Scale.pdf
>
> As per the document, they are utilizing 1100+ Kafka brokers, over 31,000
> topics, 350,000+ partitions.
>
> And as per my understanding, only a limited number of brokers participate
> in message delivery at a time for a topic.
> Since partitions are the unit of parallelism, if we have 5 partitions in a
> topic, only maximum 5 brokers can deliver the message at the same time.
> (Actually, consumers fetch the messages instead brokers deliver them, but
> anyway.)
>
> Since it's not required for all brokers to participate in all message
> processing, when we need to scale out Kafka, we can just add more brokers.
>
> For example, if we have 5 topics with 5 partitions each in 5 brokers.
> There will be 1 partition for each topic in each broker. As more and more
> topics are created, there will be more partitions in one broker.
> Let's say the maximum number of partitions that one broker properly handles
> in time is 10, then if there are 10 topics, message processing will start
> becoming slow.
> Then we can just add more brokers. If we add 1 broker, it can handle 10
> partitions and it is equal to 2 topics in this example.
> So whenever 2 new topics are created we can add a broker so that we can
> keep the number of partitions that one broker holds same.
> If we can keep the number of partitions in one broker at the same level, we
> can guarantee the performance of Kafka as one broker will only participate
> in the limited number of partition processing.
>
> In this scene, metadata sharing for partitions might be considered as a
> bottleneck.
> But once partitions are assigned to brokers, it does not change over
> time(if we don't explicitly rebalance them).
> Then consumers can just statically read the location of partitions and
> fetch data from target brokers.
>
> Partitions are evenly distributed among brokers and each consumer will
> access to a few of them only at a time, we can achieve massive scalability
> with Kafka.
>
> Since this is based on my understanding about Kafka rather than real
> experience, if I observe any issues during benchmarking I will share them
> as well.
>
>
>
> Thanks
> Regards
> Dominic
>
>
>
>
>
>
> 2018년 11월 29일 (목) 오전 9:54, Rodric Rabbah 님이 작성:
>
> > Thanks for sharing Dominic. I'll need to read this slowly to understand
> it.
> > It does look like a bit of the proposal you've discussed in the past
> with 1
> > topic per action name (per namespace). Have you seen this scale in your
> > production to thousands of topics?
> >
> > -r
> >
> > On Wed, Nov 28, 2018 at 5:59 AM Dominic Kim  wrote:
> >
> > > Dear whiskers.
> > >
> > > I`ve just submitted the proposal for the next version of Autonomous
> > > Container Scheduler.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduler+v2
> > >
> > > We(Naver), already implemented the scheduler based on that proposal and
> > > it's almost finished.
> > > It's not all-round scheduler but we could observe many performance
> > benefits
> > > in a few cases.
> > >
> > > We(Naver) are willing to contribute it and preparing for the
> > contribution.
> > > I hope the proposal would help you guys understand the implementation.
> > >
> > > To contribute the scheduler, it is required to contribute SPI for
> > > ContainerPool and ContainerProxy first.
> > > I will start working on this.
> > > And I will describe the details about each component in each subpage.
> > > Currently, they are blank.
> > >
> > > Please be noted again that the scheduler is not for all cases.
> > > But it would be useful in many cases until new implementation based on
> > the
> > > future architecture of OW would be landed.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture?src=contextnavpagetreemode
> > >
> > > Any comments and feedback would be very grateful.
> > >
> > > Thanks
> > > Regards
> > > Dominic
> > >
> >
>

Re: Logstore usage during `activation get`

[Nick Mitchell]

from a usability perspective, there is something quite attractive about
having ready access to logs (and activation records generally)... while
developing. i think most of the woes we encounter with logs come from high
volume actions (either exercised in prod, or during a high volume test)?

i realize that this gets us into a broader topic, but it would be nice if
we could at least support ready access to say the last 1MB of activation
information per user.

otherwise, we are sacrificing developer experience at the altar of service
provider bottom line. don't get me wrong, i am sympathetic with the
conundrum, but there are technical solutions that can satisfy both, if we
put our minds to it.

nick

On Thu, Oct 4, 2018 at 11:49 AM James W Dubee  wrote:

> Hey guys,
>
> For scalability concerns, I think it is preferable to separate logs from
> activation records entirely. Doing so would require users to use the logs
> API, or whatever tools are provided by the underly logging service. We
> should focus on serverless technology and delegate log handling to services
> that specialize in such areas.
>
> From experimenting with storing logs in a different data store other than
> where activations are stored, I've seen that there may be intermittent test
> failures for tests that rely on data being present in an activation
> record's log field. This may result when the activation record is retrieved
> before all the logs are stored in the separate log store.
>
> On a similar note, I don't think we should be storing activation responses
> at all.
>
> Regards,
> James Dubee
>
>
>
> [image: Inactive hide details for Rodric Rabbah ---10/02/2018 07:30:22
> PM---> By "break this" do you mean at some point we should remov]Rodric
> Rabbah ---10/02/2018 07:30:22 PM---> By "break this" do you mean at some
> point we should remove the logs from the GET?
>
> From: Rodric Rabbah 
> To: dev@openwhisk.apache.org
> Date: 10/02/2018 07:30 PM
> Subject: Re: Logstore usage during `activation get`
> --
>
>
>
> > By "break this" do you mean at some point we should remove the logs from
> the GET?
>
> Yes.
>
> @dubee thoughts? Since you've worked on the elastic plugin.
>
> -r
>
>
>
>

Re: MiniWhisk: what you think?

[Nick Mitchell]

With a recent update, the Shell can now be used in a fully headless mode.
But one may also leverage the fact that it is an Electron app, and hence
you have access to all of chrome as well.

On Wed, Jul 25, 2018, 20:11 Rodric Rabbah  wrote:

> https://github.com/ibm-functions/shell
>
> -r
>
> > On Jul 25, 2018, at 4:10 PM, Michele Sciabarra 
> wrote:
> >
> > Sorry for the stupid question but which shell?  bx ? wsk ?
> >
> > Because the build part could be implemented using existing solutions
> (there are many)
> >
> > --
> >  Michele Sciabarra
> >  openwh...@sciabarra.com
> >
> > - Original message -
> > From: Nick Mitchell 
> > To: dev@openwhisk.apache.org
> > Subject: Re: MiniWhisk: what you think?
> > Date: Wed, 25 Jul 2018 14:35:37 -0400
> >
> > the shell indeed has a `local` plugin that does most of this, though it
> > does not support the `go build` part; but it does take care of pulling
> the
> > base images, injecting code, and routes invokes to and from the
> container.
> >
> > for nodejs, we can also optionally hand the local invocation off to the
> > chrome debugger.
>

Re: MiniWhisk: what you think?

[Nick Mitchell]

the shell indeed has a `local` plugin that does most of this, though it
does not support the `go build` part; but it does take care of pulling the
base images, injecting code, and routes invokes to and from the container.

for nodejs, we can also optionally hand the local invocation off to the
chrome debugger.

Re: share additional deployment related limits through API host

[Nick Mitchell]

yes please! https://github.com/apache/incubator-openwhisk/issues/335

On Mon, Jul 9, 2018 at 12:57 PM, Rodric Rabbah  wrote:

> currently if you query the API host (GET /openwhisk) you get a response
> which includes the deployment manifest and action invocation limits. I've
> added in this PR [1] additional limits to include the min/max action
> duration, memory size and log size so the response looks like:
>
> "limits": {
> "min_action_logs": 134217728,
> "max_action_memory": 536870912,
> "actions_per_minute": 60,
> "min_action_memory": 134217728,
> "concurrent_actions": 30,
> "triggers_per_minute": 60,
> "sequence_length": 50,
> "max_action_logs": 536870912,
> "min_action_duration": 100,
> "max_action_duration": 30
>   }
>
>
> Thoughts?
>
> [1] https://github.com/apache/incubator-openwhisk/pull/3859
>

Re: Supporting user-configurable warm action containers?

[Nick Mitchell]

for nodejs at least: the cost of a few requires of common packages can
easily get you up to the 150-200ms range (e.g. request is a big hitter; and
this is all on top of the cost of starting a container!). perhaps, for
nodejs at least, there are only a few options, ultimately: user pays more
for idle resources; provider pays more for idle stem cells; or users take a
very hard line on the modules they import.

switching to other (compiled) runtimes might help, e.g. with the recent
work on precompiled go and swift actions? we'd still be left with the
container start times, but at least this is something we can control, e.g.
by requiring users to pay more for access to a larger prewarmed pool?

nick


On Thu, May 31, 2018 at 7:22 AM, James Thomas  wrote:

> One of most frequent complaints[1][2][3] I hear from developers using
> serverless platforms is coping with cold-start latency when dealing with
> sudden bursts of traffic.
>
> Developers often ask for a feature where they can set the number of warm
> containers kept in the cache for a function. This would allow them to keep
> a higher number of warm containers for applications with bursty traffic
> and/or upgrade the cached number prior to an anticpated burst of traffic
> arriving. This would be exposed by the managed platforms as a chargable
> feature.
>
> Is this something we could support on OpenWhisk? Ignoring the complexity
> and feasibility of any solution, from a developer POV I can image having an
> action annotation `max-warm` which would set the maximum number of warm
> containers allowed in the cache.
>
> Tyson is currently working on concurrent activation processing, which is
> one approach to reducing cold-start delays[4]. However, there are some
> downsides to concurrent activations, like no runtime isolation for request
> processing, which might make this feature inappropraite for some users.
>
> [1]
> https://www.reddit.com/r/aws/comments/6w1hip/how_many_
> successive_lambda_invocations_will_use_a/
> [2]
> https://twitter.com/search?f=tweets=default=%20%
> 23AWSWishlist%20warm=typd
> [3]
> https://theburningmonk.com/2018/01/im-afraid-youre-
> thinking-about-aws-lambda-cold-starts-all-wrong/
> [4] - https://github.com/apache/incubator-openwhisk/pull/2795
>
> --
> Regards,
> James Thomas
>

Re: Transactionid in the ErrorResponse

[Nick Mitchell]

it's unlikely, for sure, but e.g. there may be a typescript spec floating
around out there

On Thu, Apr 19, 2018 at 11:58 AM, Rodric Rabbah <rod...@gmail.com> wrote:

> It is changing the schema for the error response, but I'm having a hard
> time thinking of a case where a client depends on the value of the "code"
> (so that a change from int to string becomes a problem) - do you have an
> example?
>
> Renaming "code" would be more worrisome to me, but apart from the CLI, I
> suspect - with no evidence to back this up other than my intuition - that
> it's not used that heavily.
>
> -r
>
> On Thu, Apr 19, 2018 at 11:51 AM, Nick Mitchell <moose...@gmail.com>
> wrote:
>
> > this seems like a breaking API change. e.g. in nodejs `===` checks would
> > break.
> >
> > On Thu, Apr 19, 2018 at 11:37 AM, Rodric Rabbah <rod...@gmail.com>
> wrote:
> >
> > > Should we also rename “code”?
> > >
> > > I don’t see the value in using code: 0 and changing the schema should
> be
> > > fine and better in the long run.
> > >
> > > -r
> > >
> > > > On Apr 19, 2018, at 11:31 AM, Christian Bickel <apa...@cbickel.de>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm currently working on a PR which basically moves the transactionId
> > > generation from the controller to the entrypoint of the system. This is
> > the
> > > nginx or a frontdoor above.
> > > > One change in this PR is to change the format of the tid from a
> number
> > > to a String.
> > > > This works pretty well except one change, that could be seen by
> users.
> > > > If there is an error in our system, we return an error response with
> a
> > > short description and the tid. Until now the tid was a number, so the
> > value
> > > in the JSON has no quotes. With this change, the response message would
> > > change, because the tid is a String.
> > > > This means the response would change from
> > > > ```
> > > > {
> > > > "error": "This is the description",
> > > > "code": 123
> > > > }
> > > > ```
> > > > to
> > > > ```
> > > > {
> > > > "error": "This is the description",
> > > > "code": "123"
> > > > }
> > > > ```.
> > > >
> > > > Do you agree, that this change would be OK?
> > > > An alternative would be to always return a 0 and add an additional
> > field
> > > with our new tid-format.
> > > >
> > > > If there are no concerns, I'll go ahead and change the field from the
> > > number to a String.
> > > >
> > > > Greetings
> > > > Christian Bickel
> > >
> >
>

Re: How to share an action but not its code, nor parameters

[Nick Mitchell]

i've wanted, for a while now, the ability to encapsulate remote requests,
whether to other actions or to remote services, behind a Composer-like
linguistic facade.

so that, to sequence a call to any REST API, i would like to have, e.g.

composer.sequence('myOwnAction', composer.request({ url: '
https://thirdparty.com', method: 'GET', ... }))

where the argument to composer.request could be precisely that of the
nodejs request NPM. we could do something similar for other languages, i.e.
adopting their dominant schematic paradigm for expressing remote
interactions.

nick

On Wed, Feb 14, 2018 at 9:35 PM, Rodric Rabbah  wrote:

>
> > How would this work with Composer ?
>
> composer.web(‘namespace/package/name’)
>
> As an example. Olivier may have other ideas.
>
> A combinator in composer is a programmable extension; so either adding
> something specific to web actions or more generally a URL which can codegen
> into an inline HTTP request could work here.
>
> -r

Re: OpenWhisk shell tool

[Nick Mitchell]

the current release (1.1.1) does not yet expose the bashy variant. i'll do
that for 1.2.0, it's ready to go, i just haven't had a chance to include it
in the dist bundles.

for 1.1.1, you can use the `run` command from within the Shell. this will
execute a sequence of Shell commands from a file. prior to 1.1.1, we didn't
expose the `run` command even in the Shell.

we're getting there, especially with all of the great feedback we're
getting. keep it coming!!

nick
@starpit


On Fri, Aug 4, 2017 at 2:46 PM, Tyson Norris <tnor...@adobe.com.invalid>
wrote:

> I didn’t get that it can be invoked from a shell in a REPL style, thanks -
> is this possible with the current downloads?
>
> I appreciate the value of the UI (looks great btw!) - its just not clear
> how to do the 'scripted execution’ parts?
> Thanks
> Tyson
>
> > On Aug 4, 2017, at 11:22 AM, Nick Mitchell <moose...@gmail.com> wrote:
> >
> > thanks tyson!
> >
> > the tool can indeed be used in a purely bashy way, if that's helpful. so
> > e.g., the following works, assuming for a second that we call the Shell
> > `wsker` for the purpose of this email:
> >
> > wsker activation get foo
> > wsker action invoke foo
> > wsker activation get 
> >
> > these look familiar, i think? hehe, what i'm getting at is that the
> command
> > set overlaps! and you can issue one-off commands to the Shell directly
> from
> > bash, if you'd like; or in a bundle via `wsker script.txt`
> >
> > i propose that you will quickly discover the value of having the electron
> > app, including:
> >
> > 1) your command history is specific to openwhisk, so you don't have to
> > arrow up past non-whisk commands
> > 2) the command set is richer
> > 3) you get visuals for the common and important data
> > 4) no more copy-paste of that  part of wsk activation get!
> > 5) an app to itself, with its own icon and one that can be separately
> > minimized/maximized, or pinned to desktops
> >
> >
> > nick
> > @starpit
> >
> > On Fri, Aug 4, 2017 at 12:23 PM, Tyson Norris <tnor...@adobe.com.invalid
> >
> > wrote:
> >
> >> Some thoughts on this: I think one thing that would make this less
> >> confusing is introducing a REPL to the wsk CLI (or a wrapper of it)
> instead
> >> of introducing REPL as “a desktop application with a gui”. I like the
> gui,
> >> but I also think it may be adding a layer of confusion.
> >>
> >> Similarly, it should be carefully considered how commands can be
> >> consistent between REPL and non-REPL usage, e.g. for related CLI auth
> >> discussion, it would be great if:
> >> wsk auth
> >> Had the same effect as:
> >> REPL: auth
> >>
> >> The difference being that wsk auth would store credentials to disk for
> use
> >> at next wsk command run, and REPL: auth would store credentials for the
> >> length of the session only. In addition, the REPL may be able to
> >> conveniently pop up an auth dialog, where the CLI may have to ask you to
> >> load a url, but the end result would be the same.
> >>
> >> From there the notion of “REPL has a special language in addition to the
> >> CLI commands” is something easier (at least for me) to get behind, as
> long
> >> as things that are not purely session-based are added to the CLI as well
> >> (like auth flavored commands).
> >>
> >> Thanks for starting this tool, I think its useful and look forward to
> >> watching it progress!
> >>
> >> Tyson
> >>
> >>
> >>
> >>
> >>> On Aug 4, 2017, at 8:59 AM, Nick Mitchell <moose...@gmail.com> wrote:
> >>>
> >>> With the shell, one would issue `last`. Or `last foo`. With a REPL, we
> >> have
> >>> the luxury of a flexible command structure that can be tailored to the
> >> task
> >>> at hand.
> >>>
> >>> And, once you are looking at that activation, you can drill down (eg to
> >>> tree view of the sequence), or reinvoke (we can remember the input), or
> >>> reinvoke with new args, or add this action to a sequence now that you
> >> know
> >>> it works (append to myseq).
> >>>
> >>> These are all plugins, so new ideas can appear as shell commands in a
> >>> matter of minutes.
> >>>
> >>> On Aug 4, 2017 11:49, "Rodric Rabbah" <rod...@gmail.com> wrote:
> >>>
> >>>> This is a very useful discussion - thank y

Re: OpenWhisk shell tool

[Nick Mitchell]

thanks tyson!

the tool can indeed be used in a purely bashy way, if that's helpful. so
e.g., the following works, assuming for a second that we call the Shell
`wsker` for the purpose of this email:

wsker activation get foo
wsker action invoke foo
wsker activation get 

these look familiar, i think? hehe, what i'm getting at is that the command
set overlaps! and you can issue one-off commands to the Shell directly from
bash, if you'd like; or in a bundle via `wsker script.txt`

i propose that you will quickly discover the value of having the electron
app, including:

1) your command history is specific to openwhisk, so you don't have to
arrow up past non-whisk commands
2) the command set is richer
3) you get visuals for the common and important data
4) no more copy-paste of that  part of wsk activation get!
5) an app to itself, with its own icon and one that can be separately
minimized/maximized, or pinned to desktops


nick
@starpit

On Fri, Aug 4, 2017 at 12:23 PM, Tyson Norris <tnor...@adobe.com.invalid>
wrote:

> Some thoughts on this: I think one thing that would make this less
> confusing is introducing a REPL to the wsk CLI (or a wrapper of it) instead
> of introducing REPL as “a desktop application with a gui”. I like the gui,
> but I also think it may be adding a layer of confusion.
>
> Similarly, it should be carefully considered how commands can be
> consistent between REPL and non-REPL usage, e.g. for related CLI auth
> discussion, it would be great if:
> wsk auth
> Had the same effect as:
> REPL: auth
>
> The difference being that wsk auth would store credentials to disk for use
> at next wsk command run, and REPL: auth would store credentials for the
> length of the session only. In addition, the REPL may be able to
> conveniently pop up an auth dialog, where the CLI may have to ask you to
> load a url, but the end result would be the same.
>
> From there the notion of “REPL has a special language in addition to the
> CLI commands” is something easier (at least for me) to get behind, as long
> as things that are not purely session-based are added to the CLI as well
> (like auth flavored commands).
>
> Thanks for starting this tool, I think its useful and look forward to
> watching it progress!
>
> Tyson
>
>
>
>
> > On Aug 4, 2017, at 8:59 AM, Nick Mitchell <moose...@gmail.com> wrote:
> >
> > With the shell, one would issue `last`. Or `last foo`. With a REPL, we
> have
> > the luxury of a flexible command structure that can be tailored to the
> task
> > at hand.
> >
> > And, once you are looking at that activation, you can drill down (eg to
> > tree view of the sequence), or reinvoke (we can remember the input), or
> > reinvoke with new args, or add this action to a sequence now that you
> know
> > it works (append to myseq).
> >
> > These are all plugins, so new ideas can appear as shell commands in a
> > matter of minutes.
> >
> > On Aug 4, 2017 11:49, "Rodric Rabbah" <rod...@gmail.com> wrote:
> >
> >> This is a very useful discussion - thank you Rob for starting it. We
> both
> >> want and need this kind of feedback!
> >>
> >> One of the observations that you noted wrt to the shell is that it has
> "its
> >> own language". Indeed there is a language here, in the same way that the
> >> wsk CLI already offers a language for serverless programming using
> >> OpenWhisk. When the language of the CLI is limiting, what do we do as
> >> developers? I posit that we layer new languages on top --- my favorite
> >> example is "give me the last activation". At one point I polled for how
> >> many bash aliases the community has come up with for this feature!
> Recently
> >> the wsk CLI added `wsk activation get --last`. That's still too verbose
> and
> >> I continue to use my local alias for this command and I expect others
> will
> >> too. More concretely, it's too verbose when I'm developing, iterating,
> and
> >> debugging.
> >>
> >> In the same way, I've seen developers share bash scripts for automating
> >> other tasks that the current wsk CLI (or really any client) doesn't yet
> >> support. For example: deleting all assets in a namespace, or a package.
> >> These are features I expect will eventually end up in the wsk CLI. But
> the
> >> gate for rapidly experimenting with new aliases, plugins, features is
> too
> >> narrow today.
> >>
> >> The "openwhisk shell" is a way of normalizing the programming experience
> >> for serverless - it does subsume the CLI in that the wsk commands should
> >&g

Re: OpenWhisk shell tool

[Nick Mitchell]

With the shell, one would issue `last`. Or `last foo`. With a REPL, we have
the luxury of a flexible command structure that can be tailored to the task
at hand.

And, once you are looking at that activation, you can drill down (eg to
tree view of the sequence), or reinvoke (we can remember the input), or
reinvoke with new args, or add this action to a sequence now that you know
it works (append to myseq).

These are all plugins, so new ideas can appear as shell commands in a
matter of minutes.

On Aug 4, 2017 11:49, "Rodric Rabbah"  wrote:

> This is a very useful discussion - thank you Rob for starting it. We both
> want and need this kind of feedback!
>
> One of the observations that you noted wrt to the shell is that it has "its
> own language". Indeed there is a language here, in the same way that the
> wsk CLI already offers a language for serverless programming using
> OpenWhisk. When the language of the CLI is limiting, what do we do as
> developers? I posit that we layer new languages on top --- my favorite
> example is "give me the last activation". At one point I polled for how
> many bash aliases the community has come up with for this feature! Recently
> the wsk CLI added `wsk activation get --last`. That's still too verbose and
> I continue to use my local alias for this command and I expect others will
> too. More concretely, it's too verbose when I'm developing, iterating, and
> debugging.
>
> In the same way, I've seen developers share bash scripts for automating
> other tasks that the current wsk CLI (or really any client) doesn't yet
> support. For example: deleting all assets in a namespace, or a package.
> These are features I expect will eventually end up in the wsk CLI. But the
> gate for rapidly experimenting with new aliases, plugins, features is too
> narrow today.
>
> The "openwhisk shell" is a way of normalizing the programming experience
> for serverless - it does subsume the CLI in that the wsk commands should
> work inside it, but also extends the capabilities to add more syntactic
> convenience and make a workflow more fluid - some of these may not be
> readily possible or practical in a terminal. One of the benefits that I've
> experienced directly is that it makes the iterative programming experience
> and development for serverless more agile and fluid.
>
> To me, this is independent of the question of scripting for deployment,
> continuous integration, and delivery and hence the context for my "old
> school" comment - the shell can export a bash script for you, or other
> artifacts like a serverless framework manifest, for managing a deployment
> (as examples).
>
>  -r
>

Re: OpenWhisk shell tool

[Nick Mitchell]

On Fri, Aug 4, 2017 at 10:32 AM, Raymond Camden <raymondcam...@gmail.com>
wrote:

> On Fri, Aug 4, 2017 at 9:25 AM, Nick Mitchell <moose...@gmail.com> wrote:
> >> It looks like it the shell has its own language, Is it documented? I'm
> >> concerned that we're inventing a whole new language off the cuff...
> >>
> >>
> > the Shell is exactly like the `wsk` tool, in that there are commands.
> yes,
> > it is a langauge, of sorts, but they really are just commands. the Shell
> > supports all of the `wsk` commands. it augments them either to add
> > functions `wsk` does not have, or to offer an alternative syntax that is
> > (perhaps) more appealing.
>
> Can I just say though - every demo I've seen so far has confused the
> heck of out me. You didn't really answer the question on
> documentation, and I think we *really* need something there to make it
> a bit more usable. (If it already exists, cool.) It looks incredibly
> cool, but the syntax is so... I don't know... cool (grin), that I
> can't even understand it. What are the plans to explain the basics?
> Again, not the wsk commands, I get those, but in your tool it looks
> alien to me! (And I don't mean that as a negative, it looks super
> powerful, just need help grokking it).
>
>
fair points, raymond!

here is the current documentation story, which admittedly needs work.

1) there is a `help` command, which will show you the commands available in
your current context. e.g. if you have selected an action, then the `set`
command will show up in the help list. `set` can be used to CRUD
parameters. e.g. `set x=1`.

2) we have this page: https://ibm.biz/wskshell which, in addition to some
videos, has hexagons that introduce the command set. in addition to
introducing the commands, there are a few "scenario" examples, that show
how to use those commands for larger tasks. for example, there is a
scenario page on creating a web page.

more is needed, and i hope you all can help me prioritize the docs tasks.
please feel free to rudely ask for docs where they don't exist!

nick

Re: OpenWhisk shell tool

[Nick Mitchell]

Thanks rob for the great questions!


> This new shell thing seems to be its own app. How accessible is it for
> people who can't use a mouse and need to drive it via keyboard only?
>
>
it can be used entirely without a mouse. for example, to open and close the
sidecar, use the Escape key. history can be navigated either with arrow
keys, or emacs cursor navigation (Ctrl+P, ...).




> It looks like an interactive thing like Swift Playgrounds with a timeline
> history. I'm not clear if that that history list on the left is interactive
> or if you can edit any of them arbitrarily and re-run them?
>
>
there is a `history` command that lets you explore and re-execute elements
of history, as you would with e.g. bash.



> I can't work out how version control works with this tool. Do we save
> documents of the timeline that we can reload? If we do, is it a text based
> format for the saved file?
>
>
the Shell does support multi-line pasting. it also has a `run` command that
allows you to execute a sequence of Shell commands stored in a file. it
does not as of yet have an `export` command, though there have been several
thoughts in this direction!

for example, to generate the docs page ibm.biz/wskshell, we have a
`demo.wsk` file that contains the handful of commands necessary to generate
the entire site.

the demo.wsk file sits in a repo, under SCM. this is just one example of
how we're doing things. i'm sure we'll find more and better ways as we go!




> How would this work for production development with a team?
>
>
truthfully, the Shell is so new that we don't have data points to give you
a good answer on this one. can you help me, by clarifying what you think
the production requirements are? i have answers, but would like to make
sure i'm answering the right questions :) thanks!



> It looks like it the shell has its own language, Is it documented? I'm
> concerned that we're inventing a whole new language off the cuff...
>
>
the Shell is exactly like the `wsk` tool, in that there are commands. yes,
it is a langauge, of sorts, but they really are just commands. the Shell
supports all of the `wsk` commands. it augments them either to add
functions `wsk` does not have, or to offer an alternative syntax that is
(perhaps) more appealing.

e.g. a -> b -> c can be used to construct sequences, and the `let` command
can be used to name the sequence... but you can also use `wsk action create
--sequence a,b,c` if that form is more comfortable to you!




> Related, how stable is this shell language? If someone writes a script in
> OWShell, it's likely to continue to work in the future? Are we going to get
> BC breaks in scripts that people publish (if they can publish them)?
>
>
it is early, admittedly, so there are no guarantees. in the earlier days of
OpenWhisk, i recall my scripts breaking every other day as `wsk` and the
API stabilized... however, we have 689 tests at this point, so i would
consider things to be fairly stable at this point.

once (soon!) we open source it, then alternative command structures can win
by vote/merit, etc.!


It is JS only? i.e. does it exclude people like me who don't understand JS
> and write their actions in the other languages?
>
>
there is nothing javascript-specific about the Shell. the `let` syntax
harkens to javascript, but it is purely sugar. we could quite easily add a
`def` syntax for action creation, etc. the command structure is populated
entirely via a hierarchical plugin architecture. more details on that soon.


I was a bit concerned to hear someone call the export to wsk format as "old
> school" That implies that there's a plan to deprecate the current CLI and
> that this OWShell is the new way to write all OpenWhisk applications. Is
> this correct and if so, is OWShell completely scriptable via bash and
> Windows PowerShell so that it can be automated? It's neat that you can
> paste into OWShell, but that's hard to do in CI environment.
>

the Shell is indeed scriptable, either via `run` from within the shell, or
directly from bash.

as to the future, , above my pay grade, but... my goal in making the
Shell was to foster this very kind of discussion: what is the best way to
program the serverless cloud? in my experience, this requires a REPL, and
one with concise syntax, so that i can quickly and iteratively refine my
understanding of the world. i don't want to write bash scripts for tasks
that are common! bake those into a richer command structure, so that we,
ALL of us, can benefit from the group's knowledge.

the `let` command is an example of this, as is the `every` command.
creating actions is more than just `wsk action create`. there are needs for
inline/anonymous functions for simple schema aligners... periodic rules are
quite syntactically complex with `wsk`. let's capture these common cases!
this is a main point of the Shell.

beyond richer commands, i feel that cloud programming also requires some
visuals, because so much of the data these days is 

Introducing the OpenWhisk Shell

[Nick Mitchell]

Hello everyone. I'd like to announce a cool new tool we're calling the
OpenWhisk Shell.

The shell offers a CLI, with a rich and easily extensible set of commands.
For example you can write `let foo = x=>x` to create an echo action; you
can write `let public/index.html = myPage.html` and have a web page up in
one line.

The tool also offers a graphical "sidecar" for visualizing some of complex
JSON structures returned by the OpenWhisk API. We're pretty excited by the
possibilities of this hybrid CLI-UI approach, and would love to hear your
feedback.

We'll be posting a series of medium posts ver the next few days.  here's
the first:  https://medium.com/@moosevan/introducing-openwhisk-shell-
70389d5cf1f2

... and a site (generated with a few simple Shell commands) with download
links, videos, and an intro to some of the commands:
https://openwhisk.ng.bluemix.net/api/v1/web/nickm_wskng_
demos/public/index.html


nick mitchell
@starpit

Re: Setting up Feeds w/ a verification step

[Nick Mitchell]

hi dragos,

an approach we took, in this regard, was the "router action" pattern.
facebook (similarly for slack) is configured to communicate with the router
action, which will discriminate challenge invocations from normal event
flow invocations.

in the case of facebook (and slack, more recently, can be configured to
operate in this way), you (not facebook) can choose the challenge
parameter. so, this can be bound to the router action in advance.

a nice thing about this is that it allows some flexibility: re-establishing
the challenge parameter can be done without tearing down any openwhisk
assets. also, the router pattern allows for a router that fans out events,
e.g. further discriminating based on some property of the incoming event
(actually more truly as a router)

the pattern also can be generalized to the case where the event source
chooses the challenge. the challenge parameter can then be self-bound
(dynamically) as a package parameter (to the package in which the router
action sits). then, even flows can easily validate the event-flow challenge
parameter (because the pre-established parameter is a package binding, it
is readily accessible by the router action)

nick

On Tue, Jun 20, 2017 at 8:45 PM, Dragos Dascalita Haut <
ddas...@adobe.com.invalid> wrote:

> Is there a pattern for registering feeds with providers that require a
> verification action to be invoked, in order to validate the webhook ?
>
>
> I'm trying to add a new feed provider for FB to the OW catalog [1]. Some
> providers may enforce a verification step before accepting the webhook; it
> could be as simple as the provider sending a verification token that it
> expects to be sent back by that webhook. See an example at [2].
>
>
> In OW triggers and rules are separate concerns, a design I personally
> like. The issue is that when the providers want to verify something,
> there's no "default verification rule" associated with a trigger.
>
>
> The pattern I'm currently trying to implement is:
>
>   1.  When the CREATE event for the feed is called, register the new
> trigger
>   2.  Create a "verification" rule for this trigger to invoke an action
> that satisfies the provider requirements
>   3.  Invoke the provider API for registering a new webhook
>   4.  Wait for the provider to verify the webhook, and then delete the
> rule created at step 2.
>
>
> Do you have any thoughts regarding this approach ?
>
>
> Thanks,
>
> dragos
>
> [1] - https://github.com/apache/incubator-openwhisk-catalog
> [2] - https://developers.facebook.com/docs/graph-api/webhooks#callback-url
>

Re: ApacheCon 2016: t-shirt ideas?

[Nick Mitchell]

maybe a bit of caution is needed in the use of "no-ops"? amongst the first
things i did with my serverless apps is get jenkins health checks going!

you may not have a server, but you have assets that can, and so will fail.

perhaps the implication of "no-ops" is that skill set needed for serverless
ops different than for conventional ops? yes, certainly, but, not as much
as i would have expected.

don't need DBA skills? that's just DBaas, not serverless (functions as
service).

don't need Express skills? we have already seen, in a recent email/blog
post, that people want to pull all of that over into the serverless world.

don't need networking skills? ha! many of us have already spent plenty of
time with 502s --- and to no fault of openwhisk! rather, it's a fact of the
loosely coupled, highly firewalled/gateway'd world we live in.

On Thu, May 4, 2017 at 2:13 PM, Eric Na  wrote:

> or "Apps - Ops = Openwhisk" to refine it further?
>
> On Thu, May 4, 2017 at 7:04 AM, Rodric Rabbah  wrote:
>
> > I like Michael's "Serverless - by Apache"
> > I'd refine Daniel's to "Apps without Ops" or "Apps with NoOps"
> >
>

Re: concurrent requests on actions

[Nick Mitchell]

won't this only be of benefit for invocations that are mostly sleepy? e.g.
I/O-bound? because if an action uses CPU flat-out, then there is no
throughput win to be had (by increasing the parallelism of CPU-bound
processes), given the small CPU sliver that each container gets -- unless
there is a concomitant increase in concurrency, i.e. CPU slice?

if so, then my gut tells me that there are more general solutions to this
(i.e. more efficient packing of I/O-bound processes)

On Mon, May 1, 2017 at 5:36 PM, Tyson Norris  wrote:

> Thanks Markus.
>
> Can you direct me to the travis job where I can see the 40+RPS? I agree
> that is a big gap and would like to take a look - I didn’t see anything in
> https://travis-ci.org/openwhisk/openwhisk/builds/226918375 ; maybe I’m
> looking in the wrong place.
>
> I will work on putting together a PR to discuss.
>
> Thanks
> Tyson
>
>
> On May 1, 2017, at 2:22 PM, Markus Thömmes > wrote:
>
> Hi Tyson,
>
> Sounds like you did a lot of investigation here, thanks a lot for that :)
>
> Seeing the numbers, 4 RPS in the "off" case seem very odd. The Travis
> build that runs the current system as is also reaches 40+ RPS. So we'd need
> to look at a mismatch here.
>
> Other than that I'd indeed suspect a great improvement in throughput from
> your work!
>
> Implementationwise I don't have a strong opionion but it might be worth to
> discuss the details first and land your impl. once all my staging is done
> (the open PRs). That'd ease git operation. If you want to discuss your
> impl. now I suggest you send a PR to my new-containerpool branch and share
> the diff here for discussion.
>
> Cheers,
> Markus
>
> Von meinem iPhone gesendet
>
> Am 01.05.2017 um 23:16 schrieb Tyson Norris  r...@adobe.com>>:
>
> Hi Michael -
> Concurrent requests would only reuse a running/warm container for
> same-action requests. So if the action has bad/rogue behavior, it will
> limit its own throughput only, not the throughput of other actions.
>
> This is ignoring the current implementation of the activation feed, which
> I guess is susceptible to a flood of slow running activations. If those
> activations are for the same action, running concurrently should be enough
> to not starve the system for other activations (with faster actions) to be
> processed. In case they are all different actions, OR not allowed to
> execute concurrently, then in the name of quality-of-service, it may also
> be desirable to reserve some resources (i.e. separate activation feeds) for
> known-to-be-faster actions, so that fast-running actions are not penalized
> for existing alongside the slow-running actions. This would require a more
> complicated throughput test to demonstrate.
>
> Thanks
> Tyson
>
>
>
>
>
>
>
> On May 1, 2017, at 1:13 PM, Michael Marth  h...@adobe.com>> wrote:
>
> Hi Tyson,
>
> 10x more throughput, i.e. Being able to run OW at 1/10 of the cost -
> definitely worth looking into :)
>
> Like Rodric mentioned before I figured some features might become more
> complex to implement, like billing, log collection, etc. But given such a
> huge advancement on throughput that would be worth it IMHO.
> One thing I wonder about, though, is resilience against rogue actions. If
> an action is blocking (in the Node-sense, not the OW-sense), would that not
> block Node’s event loop and thus block other actions in that container? One
> could argue, though, that this rogue action would only block other
> executions of itself, not affect other actions or customers. WDYT?
>
> Michael
>
>
>
>
> On 01/05/17 17:54, "Tyson Norris"  r...@adobe.com>> wrote:
>
> Hi All -
> I created this issue some time ago to discuss concurrent requests on
> actions: [1] Some people mentioned discussing on the mailing list so I
> wanted to start that discussion.
>
> I’ve been doing some testing against this branch with Markus’s work on the
> new container pool: [2]
> I believe there are a few open PRs in upstream related to this work, but
> this seemed like a reasonable place to test against a variety of the
> reactive invoker and pool changes - I’d be interested to hear if anyone
> disagrees.
>
> Recently I ran some tests
> - with “throughput.sh” in [3] using concurrency of 10 (it will also be
> interesting to test with the --rps option in loadtest...)
> - using a change that checks actions for an annotation “max-concurrent”
> (in case there is some reason actions want to enforce current behavior of
> strict serial invocation per container?)
> - when scheduling an actions against the pool, if there is a currently
> “busy” container with this action, AND the annotation is present for this
> action, AND concurrent requests < max-concurrent, the this container is
> used to invoke the action
>
> Below is a summary (approx 10x throughput 

Re: vanity urls for web actions

[Nick Mitchell]

i would place this pretty high up on the list of requirements for
supporting serverless web apps (e.g. without it, secure cookies and local
storage are not possible).

yes, we could require a separate reverse proxy front-end, but for a couple
of lines of static nginx config, this seems like a pretty easy choice, to
me.

nick

On Fri, Feb 17, 2017 at 2:39 PM, Rodric Rabbah  wrote:

> I submitted a pull request after playing around with this locally to allow
> our edge router to rewrite URLs to provide a vanity URL for web actions in
> a namespace; it's a handful of lines to the edge nginx configuration file.
> It matches the namespace from the server name and rewrites the request to
> the fully qualified web action name.
>
> So  https://guest.openwhisk.host/public/index.html
> becomes a proxy for
>   https://openwhisk.host/api/v1/experimental/web/guest/public/
> index.html
>
>
> For added convenience, and by convention:
>   https://guest.openwhisk.host becomes a proxy for
>   https://openwhisk.host/api/v1/experimental/web/guest/public/
> index.html
>
> This of course requires DNS resolution or an entry in /etc/hosts for it to
> work locally.
>
> For example:
> > grep guest /etc/hosts 192.168.99.100  guest.dockerhost
>
> But this makes for a mich nicer URLs even for recording local demos.
>
> https://github.com/openwhisk/openwhisk/pull/1871
>
> -r
>

Re: Allow actions to be accessible from a web browser

[Nick Mitchell]

i like that this gives me a stable/predictable route, so that i don't have
to keep around a registry of endpoints in my deployment scripts -- as i
would if the routes were exposed with a random-ish hash in them.

and i like the idea of a simple way of supporting client applications that
doesn't require an extra step in deployment scripts. i.e. i just tack a `-a
export true` on to the create/update steps.

i also like the ability to do simple projections, thus avoiding the need
for `jq` postprocessing. if i project a field, do i also have to type it,
e.g. /field/x/int? or are the mime types only needed when you want to force
a non-default interpretation?

using mime types to request a particular response header is pretty awesome!
though perhaps its use may be somewhat constrained (until we have
streaming?) by any payload limitations a whisk installation might have in
place.

On Thu, Jan 12, 2017 at 5:51 PM, Markus Thömmes 
wrote:

> Haven't looked at the implementation yet but I really dig the idea!
>
> Are query parameters forwarded to the action as well?
>
> - mt
>
> Von meinem iPhone gesendet
>
> > Am 12.01.2017 um 23:44 schrieb Rodric Rabbah :
> >
> > I just opened a pull request to allow actions to be accessible viaa  web
> > browser. Action invokes this way are anonymous in that the caller is not
> > authenticated. The intended action must be named in the path as a fully
> > qualified name as in
> > /experimental/web/some-namespace/some-package/some-action. The package
> is
> > optional in that the action may be in the default package. In which case,
> > the string "default" must be used.
> >
> > If the action doesn't exist (or the namespace is not valid) a BadRequest
> is
> > generated. Optionally, the result form the action may be projected based
> on
> > a named property. As in
> > /experimental/web/some-namespace/some-package/some-action/some-property.
> If
> > the property
> > does not exist in the result then a BadRequest is generated. By
> convention,
> > the "html" property will attempt to respond with media type "text/html".
> >
> > Actions may be exposed to this web proxy by adding an annotation
> ("export"
> > -> true).
> > Demo video https://ibm.box.com/s/5c6ignvejihbai3f59uvqcxee9etf0lf.
> >
> > Feedback solicited and welcomed.
> >
> > -r
>