This is an automated email from the ASF dual-hosted git repository.
dgrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git
The following commit(s) were added to refs/heads/master by this push:
new 9c05de4 Document adding a new runtime kind. (#3837)
9c05de4 is described below
commit 9c05de4494d6edbd198fd494d458b908bbc0cd0a
Author: rodric rabbah <[email protected]>
AuthorDate: Sat Jul 7 12:35:23 2018 -0400
Document adding a new runtime kind. (#3837)
Document the steps required to add a new runtime kind to the main repo so
that it is available to the OpenWhisk deployment.
---
docs/actions-new.md | 251 ++++++++++++++++++++++++++++++++++++++++++++++++++
docs/actions-swift.md | 60 ++++++------
docs/actions.md | 15 ++-
3 files changed, 291 insertions(+), 35 deletions(-)
diff --git a/docs/actions-new.md b/docs/actions-new.md
new file mode 100644
index 0000000..7ae6b16
--- /dev/null
+++ b/docs/actions-new.md
@@ -0,0 +1,251 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+OpenWhisk supports [several languages and
runtimes](actions.md#languages-and-runtimes) but
+there may be other languages or runtimes that are important for your
organization, and for
+which you want tighter integration with the platform. The OpenWhisk platform
is extensible
+and you can add new languages or runtimes (with custom packages and
third-party dependencies)
+following the guide described here.
+
+### Unit of execution
+The unit of execution for all functions is a [Docker
container](https://docs.docker.com).
+The container implements a specific interface which:
+1. accepts an initialization payload (the code) and prepared for execution,
+2. accepts runtime payload (the input parameters) and
+ - prepares the activation context,
+ - runs the function,
+ - returns the function result,
+3. flushes all `stdout` and `stderr` logs and adds a frame marker at the end
of the activation.
+
+Any container which implements [the interface](#action-interface) may be used
as an action.
+It is in this way that you can add support for other languages or customized
runtimes.
+
+The interface is enforced via a [canonical test
suite](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+which validates the initialization protocol, the runtime protocol, ensures the
activation context is correctly prepared,
+and that the logs are properly framed. Your runtime should extend this test
suite, and of course include additional tests
+as needed.
+
+The runtime support is best implemented in its own repository to permit a
management
+lifecycle independent of the rest of the OpenWhisk platform which requires the
following
+additions:
+1. introduce the runtime specification into the [runtimes
manifest](../ansible/files/runtimes.json),
+2. add a new `actions-<your runtime>.md` file to the [docs](.) directory,
+3. add a link to your new language or runtime to the [top level
index](actions.md#languages-and-runtimes),
+4. add the runtime to the [Swagger
file](../core/controller/src/main/resources/apiv1swagger.json),
+5. add a standard test action to the [tests artifacts
directory](../tests/dat/actions/unicode.tests).
+
+### The runtime manifest
+
+Actions when created specify the desired runtime for the function via a
property called "kind".
+When using the `wsk` CLI, this is specified as `--kind <runtime-kind>`. The
value is a typically
+a string describing the language (e.g., `nodejs`) followed by a colon and the
version for the runtime
+as in `nodejs:8` or `php:7.2`.
+
+The manifest is a map of runtime family names to an array of specific kinds.
The details of the
+schema are found in the [Exec
Manifest](../common/scala/src/main/scala/whisk/core/entity/ExecManifest.scala).
+As an example, the following entry add a new runtime family called `nodejs`
with a single kind
+`nodejs:6`.
+
+```json
+{
+ "nodejs": [{
+ "kind": "nodejs:6",
+ "default": true,
+ "image": {
+ "prefix": "openwhisk",
+ "name": "nodejs6action",
+ "tag": "latest"
+ }
+ }]
+}
+```
+
+The `default` property indicates if the corresponding kind should be treated
as the
+default for the runtime family. The `image` structure defines the Docker image
name
+that is used for actions of this kind (e.g., `openwhisk/nodejs6action:latest`
here).
+
+### The test action
+
+The standard test action is shown below in JavaScript. It should be adapted
for the
+new language and added to the [test artifacts
directory](../tests/dat/actions/unicode.tests)
+with the name `<runtime-kind>.txt` for plain text file or `<runtime-kind>.bin`
for a
+a binary file. The `<runtime-kind>` must match the value used for `kind` in
the corresponding
+runtime manifest entry.
+
+```js
+function main(args) {
+ var str = args.delimiter + " ☃ " + args.delimiter;
+ console.log(str);
+ return { "winter": str };
+}
+```
+
+### Canonical runtime repository
+
+The runtime repository should follow the canonical structure used by other
runtimes.
+
+```
+/path/to/runtime
+├── build.gradle # Gradle build file to integrate with rest of build
and test framework
+├── core
+│ └── <runtime name and version>
+│ ├── Dockerfile # builds the runtime's Docker image
+│ └── ... # your runtimes files which implement the action proxy
+└── tests
+ └── src # tests suits...
+ └── ... # ... which extend canonical interface plus
additional runtime specific tests
+```
+
+The [Docker skeleton
repository](https://github.com/apache/incubator-openwhisk-runtime-docker)
+is an example starting point to fork and modify for your new runtime.
+
+### Action Interface
+
+An action consists of the user function (and its dependencies) along with a
_proxy_ that implements a
+canonical protocol to integrate with the OpenWhisk platform.
+
+The proxy is a web server with two endpoints.
+* It listens on port `8080`.
+* It implements `/init` to initialize the container.
+* It also implements `/run` to activate the function.
+
+The proxy also prepares the
+[execution
context](actions.md#accessing-action-metadata-within-the-action-body),
+and flushes the logs produced by the function to stdout and stderr.
+
+#### Initialization
+
+The initialization route is `/init`. It must accept a `POST` request with a
JSON object as follows:
+```
+{
+ "value": {
+ "name" : String,
+ "main" : String,
+ "code" : String,
+ "binary": Boolean
+ }
+}
+```
+
+* `name` is the name of the action.
+* `main` is the name of the function to execute.
+* `code` is either plain text or a base64 encoded string for binary functions
(i.e., a compiled executable).
+* `binary` is false if `code` is in plain text, and true if `code` is base64
encoded.
+
+The initialization route is called exactly once by the OpenWhisk platform,
before executing a function.
+The route should report an error if called more than once. It is possible
however that a single initialization
+will be followed by many activations (via `/run`).
+
+**Successful initialization:** The route should respond with `200 OK` if the
initialization is successful and
+the function is ready to execute. Any content provided in the response is
ignored.
+
+**Failures to initialize:** Any response other than `200 OK` is treated as an
error to initialize. The response
+from the handler if provided must be a JSON object with a single field called
`error` describing the failure.
+The value of the error field may be any valid JSON value. The proxy should
make sure to generate meaningful log
+message on failure to aid the end user in understanding the failure.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60
seconds). The initialization
+must complete within the allowed duration. Failure to complete initialization
within the allowed time frame
+will destroy the container.
+
+**Limitation:** The proxy does not currently receive any of the activation
context at initialization time.
+There are scenarios where the context is convenient if present during
initialization. This will require a
+change in the OpenWhisk platform itself. Note that even if the context is
available during initialization,
+it must be reset with every new activation since the information will change
with every execution.
+
+#### Activation
+
+The proxy is ready to execute a function once it has successfully completed
initialization. The OpenWhisk
+platform will invoke the function by posting an HTTP request to `/run` with a
JSON object providing a new
+activation context and the input parameters for the function. There may be
many activations of the same
+function against the same proxy (viz. container). Currently, the activations
are guaranteed not to overlap
+— that is, at any given time, there is at most one request to `/run` from the
OpenWhisk platform.
+
+The route must accept a JSON object and respond with a JSON object, otherwise
the OpenWhisk platform will
+treat the activation as a failure and proceed to destroy the container. The
JSON object provided by the
+platform follows the following schema:
+```
+{
+ "value": JSON,
+ "namespace": String,
+ "action_name": String,
+ "api_host": String,
+ "api_key": String,
+ "activation_id": String,
+ "deadline": Number
+}
+```
+
+* `value` is a JSON object and contains all the parameters for the function
activation.
+* `namespace` is the OpenWhisk namespace for the action (e.g., `whisk.system`).
+* `action_name` is the [fully qualified
name](reference.md#fully-qualified-names) of the action.
+* `activation_id` is a unique ID for this activation.
+* `deadline` is the deadline for the function.
+* `api_key` is the API key used to invoke the action.
+
+The `value` is the function parameters. The rest of the properties become part
of the activation context
+which is a set of environment variables constructed by capitalizing each of
the property names, and prefixing
+the result with `__OW_`. Additionally, the context must define `__OW_API_HOST`
whose value
+is the OpenWhisk API host. This value is currently provided as an environment
variable defined at container
+startup time and hence already available in the context.
+
+**Successful activation:** The route must respond with `200 OK` if the
activation is successful and
+the function has produced a JSON object as its result. The response body is
recorded as the [result
+of the activation](actions.md#understanding-the-activation-record).
+
+**Failed activation:** Any response other than `200 OK` is treated as an
activation error. The response
+from the handler must be a JSON object with a single field called `error`
describing the failure.
+The value of the error field may be any valid JSON value. Should the proxy
fail to respond with a JSON
+object, the OpenWhisk platform will treat the failure as an uncaught
exception. These two failures modes are
+distinguished by the value of the `response.status` in the [activation
record](actions.md#understanding-the-activation-record)
+which is "application error" if the proxy returned an "error" object, and
"action developer error" otherwise.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60
seconds). The activation
+must complete within the allowed duration. Failure to complete activation
within the allowed time frame
+will destroy the container.
+
+#### Logs
+
+The proxy must flush all the logs produced during initialization and execution
and add a frame marker
+to denote the end of the log stream for an activation. This is done by
emitting the token
+[`XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX`](https://github.com/apache/incubator-openwhisk/blob/59abfccf91b58ee39f184030374203f1bf372f2d/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainer.scala#L51)
+as the last log line for the `stdout` _and_ `stderr` streams. Failure to emit
this marker will cause delayed
+or truncated activation logs.
+
+### Testing the new runtime
+
+There is a [canonical test
harness](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+for validating a new runtime. The harness will performing the following:
+* Test the proxy can handle the identity functions (initialize and run).
+* Test the proxy can properly handle functions with Unicode characters.
+* Test the proxy properly constructs the activation context.
+* Test the proxy can handle large payloads (more than 1MB).
+* Test the proxy does not permit re-initialization.
+* Test the error handling for an action returning an invalid response.
+
+The canonical test suite should be extended by the new runtime tests.
Additional
+tests will be required depending on the feature set provided by the runtime.
+
+Since the OpenWhisk platform is language and runtime agnostic, it is generally
not
+necessary to add integration tests. That is the unit tests verifying the
protocol are
+sufficient. However, it may be necessary in some cases to modify the `wsk` CLI
or
+other OpenWhisk clients. In which case, appropriate tests should be added as
necessary.
+The OpenWhisk platform will perform a generic integration test as part of its
basic
+system tests. This integration test will require a [test
function](#the-test-action) to
+be available so that the test harness can create, invoke, and delete the
action.
diff --git a/docs/actions-swift.md b/docs/actions-swift.md
index 859583d..5db3687 100644
--- a/docs/actions-swift.md
+++ b/docs/actions-swift.md
@@ -100,9 +100,9 @@ wsk action invoke --result helloSwift --param name World
```
```json
- {
- "greeting": "Hello World!"
- }
+{
+ "greeting": "Hello World!"
+}
```
Find out more about parameters in the [Working with
parameters](./parameters.md) section.
@@ -120,12 +120,12 @@ You can use a script to automate the packaging of the
action. Create script `co
set -ex
if [ -z "$1" ] ; then
- echo 'Error: Missing action name'
- exit 1
+ echo 'Error: Missing action name'
+ exit 1
fi
if [ -z "$2" ] ; then
- echo 'Error: Missing kind, for example swift:4.1'
- exit 2
+ echo 'Error: Missing kind, for example swift:4.1'
+ exit 2
fi
OUTPUT_DIR="build"
if [ ${2} == "swift:3.1.1" ]; then
@@ -144,14 +144,14 @@ DEST_PACKAGE_SWIFT="$BASE_PATH/spm-build/Package.swift"
BUILD_FLAGS=""
if [ -n "$3" ] ; then
- BUILD_FLAGS=${3}
+ BUILD_FLAGS=${3}
fi
echo "Using runtime $RUNTIME to compile swift"
docker run --rm --name=compile-ow-swift -it -v "$(pwd):/owexec" $RUNTIME bash
-ex -c "
if [ -f \"/owexec/$OUTPUT_DIR/$1.zip\" ] ; then
- rm \"/owexec/$OUTPUT_DIR/$1.zip\"
+ rm \"/owexec/$OUTPUT_DIR/$1.zip\"
fi
echo 'Setting up build...'
@@ -159,8 +159,8 @@ cp /owexec/actions/$1/Sources/*.swift $DEST_SOURCE/
# action file can be either {action name}.swift or main.swift
if [ -f \"$DEST_SOURCE/$1.swift\" ] ; then
- echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
- mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
+ echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
+ mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
fi
# Add in the OW specific bits
cat $BASE_PATH/epilogue.swift >> $DEST_SOURCE/main.swift
@@ -202,14 +202,14 @@ For Swift 3 here is an example:
import PackageDescription
let package = Package(
- name: "Action",
- dependencies: [
- .Package(url:
"https://github.com/apple/example-package-deckofplayingcards.git";,
majorVersion: 3),
- .Package(url: "https://github.com/IBM-Swift/CCurl.git";, "0.2.3"),
- .Package(url: "https://github.com/IBM-Swift/Kitura-net.git";,
"1.7.10"),
- .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git";,
"15.0.1"),
- .Package(url:
"https://github.com/watson-developer-cloud/swift-sdk.git";, "0.16.0")
- ]
+ name: "Action",
+ dependencies: [
+ .Package(url:
"https://github.com/apple/example-package-deckofplayingcards.git";,
majorVersion: 3),
+ .Package(url: "https://github.com/IBM-Swift/CCurl.git";, "0.2.3"),
+ .Package(url: "https://github.com/IBM-Swift/Kitura-net.git";,
"1.7.10"),
+ .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git";,
"15.0.1"),
+ .Package(url:
"https://github.com/watson-developer-cloud/swift-sdk.git";, "0.16.0")
+ ]
)
```
For Swift 4 here is an example:
@@ -220,20 +220,20 @@ For Swift 3 here is an example:
let package = Package(
name: "Action",
products: [
- .executable(
- name: "Action",
- targets: ["Action"]
- )
+ .executable(
+ name: "Action",
+ targets: ["Action"]
+ )
],
dependencies: [
- .package(url:
"https://github.com/apple/example-package-deckofplayingcards.git";,
.upToNextMajor(from: "3.0.0"))
+ .package(url:
"https://github.com/apple/example-package-deckofplayingcards.git";,
.upToNextMajor(from: "3.0.0"))
],
targets: [
- .target(
- name: "Action",
- dependencies: ["DeckOfPlayingCards"],
- path: "."
- )
+ .target(
+ name: "Action",
+ dependencies: ["DeckOfPlayingCards"],
+ path: "."
+ )
]
)
```
@@ -281,7 +281,7 @@ enum VendingMachineError: Error {
}
func main(param: Input, completion: (Output?, Error?) -> Void) -> Void {
// Return real error
- do{
+ do {
throw VendingMachineError.insufficientFunds(coinsNeeded: 5)
} catch {
completion(nil, error)
diff --git a/docs/actions.md b/docs/actions.md
index 0097b12..f365ccd 100644
--- a/docs/actions.md
+++ b/docs/actions.md
@@ -51,16 +51,21 @@ Longer tutorials that are specific to a language of your
choice are listed below
We recommend reading the basics in this document first, which are language
agnostic, before getting deeper
into a language-specific tutorial. If your preferred language isn't supported
directly, you may find
the [Docker](docker-actions.md) action or [native
binary](docker-actions.md#creating-native-actions)
-paths more suitable. Multiple actions may be composed together to create a
longer processing pipeline called a
-[sequence](#creating-action-sequences). A more advanced form of composition is
described [here](conductors.md).
+paths more suitable. Or, you can [create a new runtime](#actions-new.md).
-* [JavaScript](actions-node.md)
-* [Python](actions-python.md)
+* [Go](actions-go.md)
* [Java](actions-java.md)
+* [JavaScript](actions-node.md)
* [PHP](actions-php.md)
+* [Python](actions-python.md)
* [Swift](actions-swift.md)
* [Docker and native binaries](actions-docker.md)
-* [Go](actions-go.md)
+
+Multiple actions from different languages may be composed together to create a
longer processing
+pipeline called a [sequence](#creating-action-sequences). The polyglot nature
of the composition is
+powerful in that it affords you the ability to use the right language for the
problem you're solving,
+and separates the orchestration of the dataflow between functions from the
choice of language.
+A more advanced form of composition is described [here](conductors.md).
## The basics
