dgrove-oss closed pull request #3837: Document adding a new runtime kind.
URL: https://github.com/apache/incubator-openwhisk/pull/3837
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/actions-new.md b/docs/actions-new.md
new file mode 100644
index 0000000000..7ae6b1641f
--- /dev/null
+++ b/docs/actions-new.md
@@ -0,0 +1,251 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+OpenWhisk supports [several languages and 
runtimes](actions.md#languages-and-runtimes) but
+there may be other languages or runtimes that are important for your 
organization, and for
+which you want tighter integration with the platform. The OpenWhisk platform 
is extensible
+and you can add new languages or runtimes (with custom packages and 
third-party dependencies)
+following the guide described here.
+
+### Unit of execution
+The unit of execution for all functions is a [Docker 
container](https://docs.docker.com).
+The container implements a specific interface which:
+1. accepts an initialization payload (the code) and prepared for execution,
+2. accepts runtime payload (the input parameters) and
+   - prepares the activation context,
+   - runs the function,
+   - returns the function result,
+3. flushes all `stdout` and `stderr` logs and adds a frame marker at the end 
of the activation.
+
+Any container which implements [the interface](#action-interface) may be used 
as an action.
+It is in this way that you can add support for other languages or customized 
runtimes.
+
+The interface is enforced via a [canonical test 
suite](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+which validates the initialization protocol, the runtime protocol, ensures the 
activation context is correctly prepared,
+and that the logs are properly framed. Your runtime should extend this test 
suite, and of course include additional tests
+as needed.
+
+The runtime support is best implemented in its own repository to permit a 
management
+lifecycle independent of the rest of the OpenWhisk platform which requires the 
following
+additions:
+1. introduce the runtime specification into the [runtimes 
manifest](../ansible/files/runtimes.json),
+2. add a new `actions-<your runtime>.md` file to the [docs](.) directory,
+3. add a link to your new language or runtime to the [top level 
index](actions.md#languages-and-runtimes),
+4. add the runtime to the [Swagger 
file](../core/controller/src/main/resources/apiv1swagger.json),
+5. add a standard test action to the [tests artifacts 
directory](../tests/dat/actions/unicode.tests).
+
+### The runtime manifest
+
+Actions when created specify the desired runtime for the function via a 
property called "kind".
+When using the `wsk` CLI, this is specified as `--kind <runtime-kind>`. The 
value is a typically
+a string describing the language (e.g., `nodejs`) followed by a colon and the 
version for the runtime
+as in `nodejs:8` or `php:7.2`.
+
+The manifest is a map of runtime family names to an array of specific kinds. 
The details of the
+schema are found in the [Exec 
Manifest](../common/scala/src/main/scala/whisk/core/entity/ExecManifest.scala).
+As an example, the following entry add a new runtime family called `nodejs` 
with a single kind
+`nodejs:6`.
+
+```json
+{
+  "nodejs": [{
+    "kind": "nodejs:6",
+    "default": true,
+    "image": {
+      "prefix": "openwhisk",
+      "name": "nodejs6action",
+      "tag": "latest"
+    }
+  }]
+}
+```
+
+The `default` property indicates if the corresponding kind should be treated 
as the
+default for the runtime family. The `image` structure defines the Docker image 
name
+that is used for actions of this kind (e.g., `openwhisk/nodejs6action:latest` 
here).
+
+### The test action
+
+The standard test action is shown below in JavaScript. It should be adapted 
for the
+new language and added to the [test artifacts 
directory](../tests/dat/actions/unicode.tests)
+with the name `<runtime-kind>.txt` for plain text file or `<runtime-kind>.bin` 
for a
+a binary file. The `<runtime-kind>` must match the value used for `kind` in 
the corresponding
+runtime manifest entry.
+
+```js
+function main(args) {
+    var str = args.delimiter + " ☃ " + args.delimiter;
+    console.log(str);
+    return { "winter": str };
+}
+```
+
+### Canonical runtime repository
+
+The runtime repository should follow the canonical structure used by other 
runtimes.
+
+```
+/path/to/runtime
+├── build.gradle         # Gradle build file to integrate with rest of build 
and test framework
+├── core
+│   └── <runtime name and version>
+│       ├── Dockerfile   # builds the runtime's Docker image
+│       └── ...          # your runtimes files which implement the action proxy
+└── tests
+    └── src              # tests suits...
+        └── ...          # ... which extend canonical interface plus 
additional runtime specific tests
+```
+
+The [Docker skeleton 
repository](https://github.com/apache/incubator-openwhisk-runtime-docker)
+is an example starting point to fork and modify for your new runtime.
+
+### Action Interface
+
+An action consists of the user function (and its dependencies) along with a 
_proxy_ that implements a
+canonical protocol to integrate with the OpenWhisk platform.
+
+The proxy is a web server with two endpoints.
+* It listens on port `8080`.
+* It implements `/init` to initialize the container.
+* It also implements `/run` to activate the function.
+
+The proxy also prepares the
+[execution 
context](actions.md#accessing-action-metadata-within-the-action-body),
+and flushes the logs produced by the function to stdout and stderr.
+
+#### Initialization
+
+The initialization route is `/init`. It must accept a `POST` request with a 
JSON object as follows:
+```
+{
+  "value": {
+    "name" : String,
+    "main" : String,
+    "code" : String,
+    "binary": Boolean
+  }
+}
+```
+
+* `name` is the name of the action.
+* `main` is the name of the function to execute.
+* `code` is either plain text or a base64 encoded string for binary functions 
(i.e., a compiled executable).
+* `binary` is false if `code` is in plain text, and true if `code` is base64 
encoded.
+
+The initialization route is called exactly once by the OpenWhisk platform, 
before executing a function.
+The route should report an error if called more than once. It is possible 
however that a single initialization
+will be followed by many activations (via `/run`).
+
+**Successful initialization:** The route should respond with `200 OK` if the 
initialization is successful and
+the function is ready to execute. Any content provided in the response is 
ignored.
+
+**Failures to initialize:** Any response other than `200 OK` is treated as an 
error to initialize. The response
+from the handler if provided must be a JSON object with a single field called 
`error` describing the failure.
+The value of the error field may be any valid JSON value. The proxy should 
make sure to generate meaningful log
+message on failure to aid the end user in understanding the failure.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 
seconds). The initialization
+must complete within the allowed duration. Failure to complete initialization 
within the allowed time frame
+will destroy the container.
+
+**Limitation:** The proxy does not currently receive any of the activation 
context at initialization time.
+There are scenarios where the context is convenient if present during 
initialization. This will require a
+change in the OpenWhisk platform itself. Note that even if the context is 
available during initialization,
+it must be reset with every new activation since the information will change 
with every execution.
+
+#### Activation
+
+The proxy is ready to execute a function once it has successfully completed 
initialization. The OpenWhisk
+platform will invoke the function by posting an HTTP request to `/run` with a 
JSON object providing a new
+activation context and the input parameters for the function. There may be 
many activations of the same
+function against the same proxy (viz. container). Currently, the activations 
are guaranteed not to overlap
+— that is, at any given time, there is at most one request to `/run` from the 
OpenWhisk platform.
+
+The route must accept a JSON object and respond with a JSON object, otherwise 
the OpenWhisk platform will
+treat the activation as a failure and proceed to destroy the container. The 
JSON object provided by the
+platform follows the following schema:
+```
+{
+  "value": JSON,
+  "namespace": String,
+  "action_name": String,
+  "api_host": String,
+  "api_key": String,
+  "activation_id": String,
+  "deadline": Number
+}
+```
+
+* `value` is a JSON object and contains all the parameters for the function 
activation.
+* `namespace` is the OpenWhisk namespace for the action (e.g., `whisk.system`).
+* `action_name` is the [fully qualified 
name](reference.md#fully-qualified-names) of the action.
+* `activation_id` is a unique ID for this activation.
+* `deadline` is the deadline for the function.
+* `api_key` is the API key used to invoke the action.
+
+The `value` is the function parameters. The rest of the properties become part 
of the activation context
+which is a set of environment variables constructed by capitalizing each of 
the property names, and prefixing
+the result with `__OW_`. Additionally, the context must define `__OW_API_HOST` 
whose value
+is the OpenWhisk API host. This value is currently provided as an environment 
variable defined at container
+startup time and hence already available in the context.
+
+**Successful activation:** The route must respond with `200 OK` if the 
activation is successful and
+the function has produced a JSON object as its result. The response body is 
recorded as the [result
+of the activation](actions.md#understanding-the-activation-record).
+
+**Failed activation:** Any response other than `200 OK` is treated as an 
activation error. The response
+from the handler must be a JSON object with a single field called `error` 
describing the failure.
+The value of the error field may be any valid JSON value. Should the proxy 
fail to respond with a JSON
+object, the OpenWhisk platform will treat the failure as an uncaught 
exception. These two failures modes are
+distinguished by the value of the `response.status` in the [activation 
record](actions.md#understanding-the-activation-record)
+which is "application error" if the proxy returned an "error" object, and 
"action developer error" otherwise.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 
seconds). The activation
+must complete within the allowed duration. Failure to complete activation 
within the allowed time frame
+will destroy the container.
+
+#### Logs
+
+The proxy must flush all the logs produced during initialization and execution 
and add a frame marker
+to denote the end of the log stream for an activation. This is done by 
emitting the token
+[`XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX`](https://github.com/apache/incubator-openwhisk/blob/59abfccf91b58ee39f184030374203f1bf372f2d/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainer.scala#L51)
+as the last log line for the `stdout` _and_ `stderr` streams. Failure to emit 
this marker will cause delayed
+or truncated activation logs.
+
+### Testing the new runtime
+
+There is a [canonical test 
harness](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+for validating a new runtime. The harness will performing the following:
+* Test the proxy can handle the identity functions (initialize and run).
+* Test the proxy can properly handle functions with Unicode characters.
+* Test the proxy properly constructs the activation context.
+* Test the proxy can handle large payloads (more than 1MB).
+* Test the proxy does not permit re-initialization.
+* Test the error handling for an action returning an invalid response.
+
+The canonical test suite should be extended by the new runtime tests. 
Additional
+tests will be required depending on the feature set provided by the runtime.
+
+Since the OpenWhisk platform is language and runtime agnostic, it is generally 
not
+necessary to add integration tests. That is the unit tests verifying the 
protocol are
+sufficient. However, it may be necessary in some cases to modify the `wsk` CLI 
or
+other OpenWhisk clients. In which case, appropriate tests should be added as 
necessary.
+The OpenWhisk platform will perform a generic integration test as part of its 
basic
+system tests. This integration test will require a [test 
function](#the-test-action) to
+be available so that the test harness can create, invoke, and delete the 
action.
diff --git a/docs/actions-swift.md b/docs/actions-swift.md
index 859583d18a..5db368737b 100644
--- a/docs/actions-swift.md
+++ b/docs/actions-swift.md
@@ -100,9 +100,9 @@ wsk action invoke --result helloSwift --param name World
 ```
 
 ```json
-  {
-      "greeting": "Hello World!"
-  }
+{
+  "greeting": "Hello World!"
+}
 ```
 
 Find out more about parameters in the [Working with 
parameters](./parameters.md) section.
@@ -120,12 +120,12 @@ You can use a script to automate the packaging of the 
action. Create  script `co
 set -ex
 
 if [ -z "$1" ] ; then
-    echo 'Error: Missing action name'
-    exit 1
+  echo 'Error: Missing action name'
+  exit 1
 fi
 if [ -z "$2" ] ; then
-    echo 'Error: Missing kind, for example swift:4.1'
-    exit 2
+  echo 'Error: Missing kind, for example swift:4.1'
+  exit 2
 fi
 OUTPUT_DIR="build"
 if [ ${2} == "swift:3.1.1" ]; then
@@ -144,14 +144,14 @@ DEST_PACKAGE_SWIFT="$BASE_PATH/spm-build/Package.swift"
 
 BUILD_FLAGS=""
 if [ -n "$3" ] ; then
-    BUILD_FLAGS=${3}
+  BUILD_FLAGS=${3}
 fi
 
 echo "Using runtime $RUNTIME to compile swift"
 docker run --rm --name=compile-ow-swift -it -v "$(pwd):/owexec" $RUNTIME bash 
-ex -c "
 
 if [ -f \"/owexec/$OUTPUT_DIR/$1.zip\" ] ; then
-    rm \"/owexec/$OUTPUT_DIR/$1.zip\"
+  rm \"/owexec/$OUTPUT_DIR/$1.zip\"
 fi
 
 echo 'Setting up build...'
@@ -159,8 +159,8 @@ cp /owexec/actions/$1/Sources/*.swift $DEST_SOURCE/
 
 # action file can be either {action name}.swift or main.swift
 if [ -f \"$DEST_SOURCE/$1.swift\" ] ; then
-    echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
-    mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
+  echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
+  mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
 fi
 # Add in the OW specific bits
 cat $BASE_PATH/epilogue.swift >> $DEST_SOURCE/main.swift
@@ -202,14 +202,14 @@ For Swift 3 here is an example:
   import PackageDescription
 
   let package = Package(
-    name: "Action",
-        dependencies: [
-            .Package(url: 
"https://github.com/apple/example-package-deckofplayingcards.git";, 
majorVersion: 3),
-            .Package(url: "https://github.com/IBM-Swift/CCurl.git";, "0.2.3"),
-            .Package(url: "https://github.com/IBM-Swift/Kitura-net.git";, 
"1.7.10"),
-            .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git";, 
"15.0.1"),
-            .Package(url: 
"https://github.com/watson-developer-cloud/swift-sdk.git";, "0.16.0")
-        ]
+      name: "Action",
+          dependencies: [
+              .Package(url: 
"https://github.com/apple/example-package-deckofplayingcards.git";, 
majorVersion: 3),
+              .Package(url: "https://github.com/IBM-Swift/CCurl.git";, "0.2.3"),
+              .Package(url: "https://github.com/IBM-Swift/Kitura-net.git";, 
"1.7.10"),
+              .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git";, 
"15.0.1"),
+              .Package(url: 
"https://github.com/watson-developer-cloud/swift-sdk.git";, "0.16.0")
+          ]
   )
   ```
   For Swift 4 here is an example:
@@ -220,20 +220,20 @@ For Swift 3 here is an example:
   let package = Package(
       name: "Action",
       products: [
-        .executable(
-          name: "Action",
-          targets:  ["Action"]
-        )
+          .executable(
+              name: "Action",
+              targets:  ["Action"]
+          )
       ],
       dependencies: [
-        .package(url: 
"https://github.com/apple/example-package-deckofplayingcards.git";, 
.upToNextMajor(from: "3.0.0"))
+          .package(url: 
"https://github.com/apple/example-package-deckofplayingcards.git";, 
.upToNextMajor(from: "3.0.0"))
       ],
       targets: [
-        .target(
-          name: "Action",
-          dependencies: ["DeckOfPlayingCards"],
-          path: "."
-        )
+          .target(
+              name: "Action",
+              dependencies: ["DeckOfPlayingCards"],
+              path: "."
+          )
       ]
   )
   ```
@@ -281,7 +281,7 @@ enum VendingMachineError: Error {
 }
 func main(param: Input, completion: (Output?, Error?) -> Void) -> Void {
     // Return real error
-    do{
+    do {
         throw VendingMachineError.insufficientFunds(coinsNeeded: 5)
     } catch {
         completion(nil, error)
diff --git a/docs/actions.md b/docs/actions.md
index 0097b12208..f365ccda73 100644
--- a/docs/actions.md
+++ b/docs/actions.md
@@ -51,16 +51,21 @@ Longer tutorials that are specific to a language of your 
choice are listed below
 We recommend reading the basics in this document first, which are language 
agnostic, before getting deeper
 into a language-specific tutorial. If your preferred language isn't supported 
directly, you may find
 the [Docker](docker-actions.md) action or [native 
binary](docker-actions.md#creating-native-actions)
-paths more suitable. Multiple actions may be composed together to create a 
longer processing pipeline called a
-[sequence](#creating-action-sequences). A more advanced form of composition is 
described [here](conductors.md).
+paths more suitable. Or, you can [create a new runtime](#actions-new.md).
 
-* [JavaScript](actions-node.md)
-* [Python](actions-python.md)
+* [Go](actions-go.md)
 * [Java](actions-java.md)
+* [JavaScript](actions-node.md)
 * [PHP](actions-php.md)
+* [Python](actions-python.md)
 * [Swift](actions-swift.md)
 * [Docker and native binaries](actions-docker.md)
-* [Go](actions-go.md)
+
+Multiple actions from different languages may be composed together to create a 
longer processing
+pipeline called a [sequence](#creating-action-sequences). The polyglot nature 
of the composition is
+powerful in that it affords you the ability to use the right language for the 
problem you're solving,
+and separates the orchestration of the dataflow between functions from the 
choice of language.
+A more advanced form of composition is described [here](conductors.md).
 
 ## The basics
 


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to