Re: How to share an action but not its code, nor parameters

[Dragos Dascalita Haut]

"... pursuing the fine grained rights approach. ..."


I was also re-reading your email and I was thinking... it would actually be v 
nice to have this in openwhisk. In this way developers can fully control what 
they want to do with the action. Another use-case I once had was to share an 
action that exposes a key-value store; in that case the action would have the 
credentials to talk to the key-value store which I didn't want to share with 
others. Having an option to hide default params from a shared actions would 
have been handy. So a fine-grained control would help other use-cases too.


dragos

From: Rodric Rabbah 
Sent: Wednesday, February 14, 2018 6:39:04 PM
To: dev@openwhisk.apache.org
Subject: Re: How to share an action but not its code, nor parameters

It’s worth mentioning that going through the web action path limits the 
compositions to those that can complete is 60s. So there are other benefits to 
pursuing the fine grained rights approach.

-r

Re: How to share an action but not its code, nor parameters

[Nick Mitchell]

i've wanted, for a while now, the ability to encapsulate remote requests,
whether to other actions or to remote services, behind a Composer-like
linguistic facade.

so that, to sequence a call to any REST API, i would like to have, e.g.

composer.sequence('myOwnAction', composer.request({ url: '
https://thirdparty.com', method: 'GET', ... }))

where the argument to composer.request could be precisely that of the
nodejs request NPM. we could do something similar for other languages, i.e.
adopting their dominant schematic paradigm for expressing remote
interactions.

nick

On Wed, Feb 14, 2018 at 9:35 PM, Rodric Rabbah  wrote:

>
> > How would this work with Composer ?
>
> composer.web(‘namespace/package/name’)
>
> As an example. Olivier may have other ideas.
>
> A combinator in composer is a programmable extension; so either adding
> something specific to web actions or more generally a URL which can codegen
> into an inline HTTP request could work here.
>
> -r

Re: How to share an action but not its code, nor parameters

[Rodric Rabbah]

It’s worth mentioning that going through the web action path limits the 
compositions to those that can complete is 60s. So there are other benefits to 
pursuing the fine grained rights approach. 

-r

Re: How to share an action but not its code, nor parameters

[Rodric Rabbah]

> How would this work with Composer ?

composer.web(‘namespace/package/name’)

As an example. Olivier may have other ideas. 

A combinator in composer is a programmable extension; so either adding 
something specific to web actions or more generally a URL which can codegen 
into an inline HTTP request could work here. 

-r

Re: How to share an action but not its code, nor parameters

[Dragos Dascalita Haut]

"...The obvious path with composer is to create a combinator for invoking a 
URL, which could treat web actions intrinsically"

How would this work with Composer ?


I.e a 3rd party developer wants to author a composition like:

compose.sequence( "/other-namespace/pkg/action1", 
"/other-namespace/pkg/action2")


The owner of "other-namespace" doesn't want to share the src for action1 or 
action2, but wants others to be able to consume those actions.


Thanks,

dragos


From: Rodric Rabbah 
Sent: Wednesday, February 14, 2018 6:10:47 PM
To: dev@openwhisk.apache.org
Subject: Re: How to share an action but not its code, nor parameters

The obvious path with composer is to create a combinator for invoking a URL, 
which could treat web actions intrinsically.

This discussion has come up before - there’s also this related open issue on 
the topic [1]. An alternative would be fine grained privileges on assets where 
the default rights today are CRUD+Activate.

[1] 
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fissues%2F2155=02%7C01%7Cddascal%40adobe.com%7C27e9ad9468524993bd2508d5741957a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636542574549914334=9mHPq6tunQqMsLWJlJwEgHgHX36kHCfyskfHgotGuVo%3D=0

-r

Re: How to share an action but not its code, nor parameters

[Rodric Rabbah]

The obvious path with composer is to create a combinator for invoking a URL, 
which could treat web actions intrinsically.

This discussion has come up before - there’s also this related open issue on 
the topic [1]. An alternative would be fine grained privileges on assets where 
the default rights today are CRUD+Activate.

[1] https://github.com/apache/incubator-openwhisk/issues/2155

-r

How to share an action but not its code, nor parameters

[Dragos Dascalita Haut]

I think some openwhiskers have this use-case and I'm thinking to start a thread 
to see what are the current options. and see if we can land a PR if we want to 
support this use-case.


The need is to share an action with 3rd party developers, while retaining the 
intellectual property of the code, and potentially the associated default 
parameters. 3rd party developers should be able to use the action in any 
composition (meaning to be able to invoke the action from other actions), but 
should not be able to read its code, nor its default params. The action should 
look like a blackbox from the outside. Only the namespace owners should be able 
to manage it ( create, update, delete, etc ).


It looks like this could be achieved in Openwhisk through web actions. The 
action becomes publicly accessible to any 3rd party developer that knows the 
URL, while the code and default params are not visible at all.


With Composer [1], we have a nice programming model, and actions can be easily 
referenced in compositions by their name. We can think of the name of the 
action ( //) as the "URI" in the context of 
OpenWhisk. So the question is if there's something today that allows us to 
reference an action from another namespace in a composition, w/o having to 
share the code with other developers. And if this is not possible, then WDYT if 
we make it possible ?


Thanks,

dragos


[1] - https://github.com/ibm-functions/composer

Re: OW Tech Interchange video and notes posted

[Michele Sciabarra]

Great.  I introduced myself and the only trace of me in the video is some buzz 
of someone in the background.  I think I will do better to try again next time.

-- 
  Michele Sciabarra
  mich...@sciabarra.com

On Wed, Feb 14, 2018, at 6:44 PM, Matt Rutkowski wrote:
> Thanks Martin for moderating! 
> 
> YouTube: https://youtu.be/Dk9dx67bRCQ
> CWIKI: 
> https://cwiki.apache.org/confluence/display/OPENWHISK/2018-02-14+OW+Tech+Interchange+-+Meeting+Notes
> 
> Vadim (via Martin) volunteered for next meeting 2018-02-28
> 
> Kind regards,
> Matt 
> 

OW Tech Interchange video and notes posted

[Matt Rutkowski]

Thanks Martin for moderating! 

YouTube: https://youtu.be/Dk9dx67bRCQ
CWIKI: 
https://cwiki.apache.org/confluence/display/OPENWHISK/2018-02-14+OW+Tech+Interchange+-+Meeting+Notes

Vadim (via Martin) volunteered for next meeting 2018-02-28

Kind regards,
Matt 

OW Tech Interchange web conf. starting in 5 mins... https://zoom.us/my/asfopenwhisk

[Matt Rutkowski]

Apache OpenWhisk, bi-weekly, Tech. Interchange web conf. is in 5 mins.
https://zoom.us/my/asfopenwhisk

Kind regards,
Matt 

Re: Query around subjects/identities view

[Rodric Rabbah]

This document explains it:
http://docs.couchdb.org/en/2.0.0/couchapp/views/joins.html
It's the equivalent of joins where the namespaces match if we were using a
relational store... which one day we should consider instead.

-r

On Wed, Feb 14, 2018 at 10:04 AM, Chetan Mehrotra  wrote:

> Thanks Rodric for the info. Can you share any CouchDB doc which
> provide more details on such a usage? Also where in OpenWhisk code
> this view document is used? In all cases seen so far query is done
> with includeDocs=true
> Chetan Mehrotra
>
>
> On Wed, Feb 14, 2018 at 8:26 PM, Rodric Rabbah  wrote:
> > The use of _id allows another document to be inlined. Here, the
> concurrency limits are added as an attachment in a separate document and
> the view allows coalescing.
> >
> > -r
>

Re: Query around subjects/identities view

[Chetan Mehrotra]

Thanks Rodric for the info. Can you share any CouchDB doc which
provide more details on such a usage? Also where in OpenWhisk code
this view document is used? In all cases seen so far query is done
with includeDocs=true
Chetan Mehrotra


On Wed, Feb 14, 2018 at 8:26 PM, Rodric Rabbah  wrote:
> The use of _id allows another document to be inlined. Here, the concurrency 
> limits are added as an attachment in a separate document and the view allows 
> coalescing.
>
> -r

Re: Query around subjects/identities view

[Rodric Rabbah]

The use of _id allows another document to be inlined. Here, the concurrency 
limits are added as an attachment in a separate document and the view allows 
coalescing. 

-r

Query around subjects/identities view

[Chetan Mehrotra]

Currently the subjects/identities view is implemented as below [1].
The 2 usages of this view are in

1. Identity#get - by namespace
2. Identity#get - by uuid, key

In both cases includeDocs is set to true. However currently the view
rendered has 2 types of documents

1. {namespace, uuid, key}
2. {_id: namespace + '/limit', namespace, uuid, key}

Are the view docs used for any purpose? If yes then whats the
significance of the '_id' field having '/limit' as suffix?

Chetan Mehrotra

[1] function (doc) {
  if(doc.uuid && doc.key && !doc.blocked) {
var v = {namespace: doc.subject, uuid: doc.uuid, key: doc.key};
emit([doc.subject], v);
emit([doc.uuid, doc.key], v);
  }
  if(doc.namespaces && !doc.blocked) {
doc.namespaces.forEach(function(namespace) {
  var v = {_id: namespace.name + '/limits', namespace:
namespace.name, uuid: namespace.uuid, key: namespace.key};
  emit([namespace.name], v);
  emit([namespace.uuid, namespace.key], v);
});
  }
}

Re: First Implementation of the self-replacing Go support for OpenWhisk

[Michele Sciabarra]

On Wed, Feb 14, 2018, at 10:45 AM, James Thomas wrote:
> Michele,
> 
> Great work on getting an implemention working so quickly... I've been
> playing with it locally and it works perfectly. 
Yes.  It is actually in the state where I can run some performance tests.
That is my next step. I want to be sure it is actually more efficient than the 
"launch an executable" approach.
It should be, but better have some numbers...
Also I still have to integrate the rest of your code supporting the various 
environment variables...

> I have a couple of
> questions about the error handling and exec process.
> 
> 1. Notifying user when exec fails?
Hmm if the exec fails, the server restarts. Probably I should send an error 
message if it fails, your right...

I want also include a "validation" step. Trying to execute once with a special 
parameter asking for the version, and execute the replacement if the version is 
compatible (so basically the user used the right library).


> 
> Looking at the code, the exec to replace the binary happens after the
> `/init` HTTP response is returned. (I assume this is because the
> socket would be closed otherwise?)
> https://github.com/sciabarracom/openwhisk-runtime-go/blob/master/openwhisk/initHandler.go#L71
Actually I had some problems here... I had to Flush the request to avoid an 
error when you where uploading.

> 
> If the exec fails, the library logs out a mesage to the console but
> this won't be available to the end user IIRC. I think it's important
> to surface this error to the user. One of the main issues people face
> with serverless is the pain in debugging.
Yup, will do that. And also implement a validation.

> 
> If someone compiles the binary for the wrong platform or makes another
> mistake building the deployment  archive, it will fail silently and
> then return the default response on run. Could we store this error
> message internally and return in the default `run` response if that
> happens?
> 
totally agree.

> 2. Can you explain how the HTTP socket is managed when replacing the
> binary? Does it get automatically closed and then re-opened by the new
> process? I can't find enough information in the exec man page to help
> me understand what happens in this instance.
That is basically a kernel stuff. When you exec a new process, all the files 
are closed.

> 
> 2.1 If the socket does get closed and re-opened, they will be a tiny
> amount of time when the HTTP server is not available. It'd be good to
> hear from someone with more experience in the platform that there's no
> way this race condition could be triggered, i.e. a `/run` request is
> fired before HTTP service is available again.
There is a race condition yes, but here we are replacing the implementation.
I have no idea how OpenWhisk handle this but some mentioned in the discussion 
the invoker can support this situation with a retry... Can someone tell us the 
behaviour of the Invoker ?

> 
> I'm looking forward to trying out this binary runtime with other languages!

Actually, to support more languages I thought there could be another way to do 
that:  pipes.

We could start another process as a child and read and write in input/output to 
communicate with the process.

This solution could solve the race issue and generalize the approach.


> 
> On 13 February 2018 at 22:07, Michele Sciabarra  
> wrote:
> > As promised I released a first implementation of Go support using the 
> > technique I described before.
> >
> > In short, a library implementing the proxy and serving both /run and /init, 
> > with the ability of replace itself with a new version.
> >
> > Using the library, implementing a function in Go looks like this:
> >
> > ---
> > package main
> >
> > import (
> > "encoding/json"
> > "fmt"
> >
> > "github.com/sciabarracom/openwhisk-runtime-go/openwhisk"
> > )
> >
> > func hello(event json.RawMessage) (json.RawMessage, error) {
> > // input and output
> > var input struct{ Name string }
> > var output struct {
> > Greetings string `json:"greetings"`
> > }
> > // read the input event
> > json.Unmarshal(event, )
> > if input.Name != "" {
> > // handle the event
> > output.Greetings = "Hello, " + input.Name
> > fmt.Println(output.Greetings)
> > return json.Marshal(output)
> > }
> > return nil, fmt.Errorf("no name specified")
> > }
> >
> > func main() {
> > openwhisk.Start(ciao)
> > }
> > ---
> >
> > Actually in practice it is better to place the function in a separate 
> > package for implementing some tests, because apparently adding tests in the 
> > main package does not work.
> >
> > Source code of the library is here:
> >
> > https://github.com/sciabarracom/openwhisk-runtime-go
> >
> > Here is a simple transcription of how it works and how I tested it.
> >
> > First you build  a 

Re: First Implementation of the self-replacing Go support for OpenWhisk

[Rodric Rabbah]

- on notifying the caller of a failed exec: logs should be available even if 
container fails between init and run. As long as the logs appear on stdout and 
stderr they are fetched and amended to the activation record. This would appear 
as a generic error message that’s system generated and inspection of the logs 
should show the rest; your idea of stashing the error to construct a better 
message could also work but I’m not sure it’s worth the added complexity.

- as I noted on a previous related thread about this implementation path, the 
race on /run may require some attention from the invoker. In particular the 
connection in the past was retried, but, was removed because retries violate 
the invariant assumed by the invoker that after init, the container is ready. 
See this pr for related patch: 
https://github.com/apache/incubator-openwhisk/pull/1888

-r

Re: First Implementation of the self-replacing Go support for OpenWhisk

[James Thomas]

Michele,

Great work on getting an implemention working so quickly... I've been
playing with it locally and it works perfectly. I have a couple of
questions about the error handling and exec process.

1. Notifying user when exec fails?

Looking at the code, the exec to replace the binary happens after the
`/init` HTTP response is returned. (I assume this is because the
socket would be closed otherwise?)
https://github.com/sciabarracom/openwhisk-runtime-go/blob/master/openwhisk/initHandler.go#L71

If the exec fails, the library logs out a mesage to the console but
this won't be available to the end user IIRC. I think it's important
to surface this error to the user. One of the main issues people face
with serverless is the pain in debugging.

If someone compiles the binary for the wrong platform or makes another
mistake building the deployment  archive, it will fail silently and
then return the default response on run. Could we store this error
message internally and return in the default `run` response if that
happens?

2. Can you explain how the HTTP socket is managed when replacing the
binary? Does it get automatically closed and then re-opened by the new
process? I can't find enough information in the exec man page to help
me understand what happens in this instance.

2.1 If the socket does get closed and re-opened, they will be a tiny
amount of time when the HTTP server is not available. It'd be good to
hear from someone with more experience in the platform that there's no
way this race condition could be triggered, i.e. a `/run` request is
fired before HTTP service is available again.

I'm looking forward to trying out this binary runtime with other languages!

On 13 February 2018 at 22:07, Michele Sciabarra  wrote:
> As promised I released a first implementation of Go support using the 
> technique I described before.
>
> In short, a library implementing the proxy and serving both /run and /init, 
> with the ability of replace itself with a new version.
>
> Using the library, implementing a function in Go looks like this:
>
> ---
> package main
>
> import (
> "encoding/json"
> "fmt"
>
> "github.com/sciabarracom/openwhisk-runtime-go/openwhisk"
> )
>
> func hello(event json.RawMessage) (json.RawMessage, error) {
> // input and output
> var input struct{ Name string }
> var output struct {
> Greetings string `json:"greetings"`
> }
> // read the input event
> json.Unmarshal(event, )
> if input.Name != "" {
> // handle the event
> output.Greetings = "Hello, " + input.Name
> fmt.Println(output.Greetings)
> return json.Marshal(output)
> }
> return nil, fmt.Errorf("no name specified")
> }
>
> func main() {
> openwhisk.Start(ciao)
> }
> ---
>
> Actually in practice it is better to place the function in a separate package 
> for implementing some tests, because apparently adding tests in the main 
> package does not work.
>
> Source code of the library is here:
>
> https://github.com/sciabarracom/openwhisk-runtime-go
>
> Here is a simple transcription of how it works and how I tested it.
>
> First you build  a couple of executable, and for simplicity you also prepare 
> the json payload for the init.
>
> $ cd test
> $ go build -o hello ../main/hello.go
> $ go build -o ciao ../main/ciao.go
> $ echo '{"value":{"binary":true,"code":"'$(base64 hello)'"}}' >hello.json
> $ echo '{"value":{"binary":true,"code":"'$(base64 ciao)'"}}' >ciao.json
>
> Now you can start the actual server
>
> $ go run ../main/exec.go
>
> Now the magic happens.
>
> Default behaviour (no executable)
>
> ```
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"error":"the action failed to locate a binary"}
> ```
>
> Now post the `hello` handler and run it:
>
> ```
> $ curl -XPOST http://localhost:8080/init -d @hello.json
> OK
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"greetings":"Hello, Mike"}
> ```
>
> As you can see, the function changed and now it implements the "hello" 
> handler.
>
> But the replaced server is still able to run init so let's do it again, 
> replacing with the "ciao" handler.
>
>
> ```
> $ curl -XPOST http://localhost:8080/init -d @ciao.json
> OK
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"saluti":"Ciao, Mike"}
> ```
>
> ---
>
> I want to say "thank you" to James Thomas, I basically copied his code for 
> implementing the /run handler since it was ready, then I added my code for 
> the /init. I also had to dig into the internals of the http Go package more 
> than I expected.
>
> Work is not yet complete, I have to manage the environment variables 
> (stealing more code from James Thomas implementation), package in Docker, 
> integrate in OpenWhisk. But definitely I believe looks like a promising start.
>
> --
>   Michele 

Tech Interchange Meeting Today

[Martin Henke]

This is another kind reminder for our bi-weekly Tech Interchange Meeting today.

The (updated) agenda up to now is:

- Introduction of new attendees
- Asking around for notable changes/updates
- Updates on the graduation / release mgmt. process for OpenWhisk by Vincent S 
Hou [20 min]
- License header auditing in the release process by Ying Chun Guo [10 min]
- Sharding Loadbalancer changes by Markus Thömmes [10 min]
- Find and confirm moderator for next meeting Feb 28th  

@all: Please contact me if you have another topic that should show up on the 
agenda.
@Matt: Can you please add the agenda  to the CWiki.

Details:
What: Apache OpenWhisk "Tech. Interchange" (bi-weekly) Zoom Meeting 
When: @ 11:00am EDT, 8am PDT, 3pm GMT, 5pm CEST , 3pm UTC 
Where: https://zoom.us/my/asfopenwhisk

Regards,
Martin