dgrove-oss closed pull request #3837: Document adding a new runtime kind. URL: https://github.com/apache/incubator-openwhisk/pull/3837
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/actions-new.md b/docs/actions-new.md new file mode 100644 index 0000000000..7ae6b1641f --- /dev/null +++ b/docs/actions-new.md @@ -0,0 +1,251 @@ +<!-- +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--> + +OpenWhisk supports [several languages and runtimes](actions.md#languages-and-runtimes) but +there may be other languages or runtimes that are important for your organization, and for +which you want tighter integration with the platform. The OpenWhisk platform is extensible +and you can add new languages or runtimes (with custom packages and third-party dependencies) +following the guide described here. + +### Unit of execution +The unit of execution for all functions is a [Docker container](https://docs.docker.com). +The container implements a specific interface which: +1. accepts an initialization payload (the code) and prepared for execution, +2. accepts runtime payload (the input parameters) and + - prepares the activation context, + - runs the function, + - returns the function result, +3. flushes all `stdout` and `stderr` logs and adds a frame marker at the end of the activation. + +Any container which implements [the interface](#action-interface) may be used as an action. +It is in this way that you can add support for other languages or customized runtimes. + +The interface is enforced via a [canonical test suite](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala) +which validates the initialization protocol, the runtime protocol, ensures the activation context is correctly prepared, +and that the logs are properly framed. Your runtime should extend this test suite, and of course include additional tests +as needed. + +The runtime support is best implemented in its own repository to permit a management +lifecycle independent of the rest of the OpenWhisk platform which requires the following +additions: +1. introduce the runtime specification into the [runtimes manifest](../ansible/files/runtimes.json), +2. add a new `actions-<your runtime>.md` file to the [docs](.) directory, +3. add a link to your new language or runtime to the [top level index](actions.md#languages-and-runtimes), +4. add the runtime to the [Swagger file](../core/controller/src/main/resources/apiv1swagger.json), +5. add a standard test action to the [tests artifacts directory](../tests/dat/actions/unicode.tests). + +### The runtime manifest + +Actions when created specify the desired runtime for the function via a property called "kind". +When using the `wsk` CLI, this is specified as `--kind <runtime-kind>`. The value is a typically +a string describing the language (e.g., `nodejs`) followed by a colon and the version for the runtime +as in `nodejs:8` or `php:7.2`. + +The manifest is a map of runtime family names to an array of specific kinds. The details of the +schema are found in the [Exec Manifest](../common/scala/src/main/scala/whisk/core/entity/ExecManifest.scala). +As an example, the following entry add a new runtime family called `nodejs` with a single kind +`nodejs:6`. + +```json +{ + "nodejs": [{ + "kind": "nodejs:6", + "default": true, + "image": { + "prefix": "openwhisk", + "name": "nodejs6action", + "tag": "latest" + } + }] +} +``` + +The `default` property indicates if the corresponding kind should be treated as the +default for the runtime family. The `image` structure defines the Docker image name +that is used for actions of this kind (e.g., `openwhisk/nodejs6action:latest` here). + +### The test action + +The standard test action is shown below in JavaScript. It should be adapted for the +new language and added to the [test artifacts directory](../tests/dat/actions/unicode.tests) +with the name `<runtime-kind>.txt` for plain text file or `<runtime-kind>.bin` for a +a binary file. The `<runtime-kind>` must match the value used for `kind` in the corresponding +runtime manifest entry. + +```js +function main(args) { + var str = args.delimiter + " ☃ " + args.delimiter; + console.log(str); + return { "winter": str }; +} +``` + +### Canonical runtime repository + +The runtime repository should follow the canonical structure used by other runtimes. + +``` +/path/to/runtime +├── build.gradle # Gradle build file to integrate with rest of build and test framework +├── core +│ └── <runtime name and version> +│ ├── Dockerfile # builds the runtime's Docker image +│ └── ... # your runtimes files which implement the action proxy +└── tests + └── src # tests suits... + └── ... # ... which extend canonical interface plus additional runtime specific tests +``` + +The [Docker skeleton repository](https://github.com/apache/incubator-openwhisk-runtime-docker) +is an example starting point to fork and modify for your new runtime. + +### Action Interface + +An action consists of the user function (and its dependencies) along with a _proxy_ that implements a +canonical protocol to integrate with the OpenWhisk platform. + +The proxy is a web server with two endpoints. +* It listens on port `8080`. +* It implements `/init` to initialize the container. +* It also implements `/run` to activate the function. + +The proxy also prepares the +[execution context](actions.md#accessing-action-metadata-within-the-action-body), +and flushes the logs produced by the function to stdout and stderr. + +#### Initialization + +The initialization route is `/init`. It must accept a `POST` request with a JSON object as follows: +``` +{ + "value": { + "name" : String, + "main" : String, + "code" : String, + "binary": Boolean + } +} +``` + +* `name` is the name of the action. +* `main` is the name of the function to execute. +* `code` is either plain text or a base64 encoded string for binary functions (i.e., a compiled executable). +* `binary` is false if `code` is in plain text, and true if `code` is base64 encoded. + +The initialization route is called exactly once by the OpenWhisk platform, before executing a function. +The route should report an error if called more than once. It is possible however that a single initialization +will be followed by many activations (via `/run`). + +**Successful initialization:** The route should respond with `200 OK` if the initialization is successful and +the function is ready to execute. Any content provided in the response is ignored. + +**Failures to initialize:** Any response other than `200 OK` is treated as an error to initialize. The response +from the handler if provided must be a JSON object with a single field called `error` describing the failure. +The value of the error field may be any valid JSON value. The proxy should make sure to generate meaningful log +message on failure to aid the end user in understanding the failure. + +**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 seconds). The initialization +must complete within the allowed duration. Failure to complete initialization within the allowed time frame +will destroy the container. + +**Limitation:** The proxy does not currently receive any of the activation context at initialization time. +There are scenarios where the context is convenient if present during initialization. This will require a +change in the OpenWhisk platform itself. Note that even if the context is available during initialization, +it must be reset with every new activation since the information will change with every execution. + +#### Activation + +The proxy is ready to execute a function once it has successfully completed initialization. The OpenWhisk +platform will invoke the function by posting an HTTP request to `/run` with a JSON object providing a new +activation context and the input parameters for the function. There may be many activations of the same +function against the same proxy (viz. container). Currently, the activations are guaranteed not to overlap +— that is, at any given time, there is at most one request to `/run` from the OpenWhisk platform. + +The route must accept a JSON object and respond with a JSON object, otherwise the OpenWhisk platform will +treat the activation as a failure and proceed to destroy the container. The JSON object provided by the +platform follows the following schema: +``` +{ + "value": JSON, + "namespace": String, + "action_name": String, + "api_host": String, + "api_key": String, + "activation_id": String, + "deadline": Number +} +``` + +* `value` is a JSON object and contains all the parameters for the function activation. +* `namespace` is the OpenWhisk namespace for the action (e.g., `whisk.system`). +* `action_name` is the [fully qualified name](reference.md#fully-qualified-names) of the action. +* `activation_id` is a unique ID for this activation. +* `deadline` is the deadline for the function. +* `api_key` is the API key used to invoke the action. + +The `value` is the function parameters. The rest of the properties become part of the activation context +which is a set of environment variables constructed by capitalizing each of the property names, and prefixing +the result with `__OW_`. Additionally, the context must define `__OW_API_HOST` whose value +is the OpenWhisk API host. This value is currently provided as an environment variable defined at container +startup time and hence already available in the context. + +**Successful activation:** The route must respond with `200 OK` if the activation is successful and +the function has produced a JSON object as its result. The response body is recorded as the [result +of the activation](actions.md#understanding-the-activation-record). + +**Failed activation:** Any response other than `200 OK` is treated as an activation error. The response +from the handler must be a JSON object with a single field called `error` describing the failure. +The value of the error field may be any valid JSON value. Should the proxy fail to respond with a JSON +object, the OpenWhisk platform will treat the failure as an uncaught exception. These two failures modes are +distinguished by the value of the `response.status` in the [activation record](actions.md#understanding-the-activation-record) +which is "application error" if the proxy returned an "error" object, and "action developer error" otherwise. + +**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 seconds). The activation +must complete within the allowed duration. Failure to complete activation within the allowed time frame +will destroy the container. + +#### Logs + +The proxy must flush all the logs produced during initialization and execution and add a frame marker +to denote the end of the log stream for an activation. This is done by emitting the token +[`XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX`](https://github.com/apache/incubator-openwhisk/blob/59abfccf91b58ee39f184030374203f1bf372f2d/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainer.scala#L51) +as the last log line for the `stdout` _and_ `stderr` streams. Failure to emit this marker will cause delayed +or truncated activation logs. + +### Testing the new runtime + +There is a [canonical test harness](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala) +for validating a new runtime. The harness will performing the following: +* Test the proxy can handle the identity functions (initialize and run). +* Test the proxy can properly handle functions with Unicode characters. +* Test the proxy properly constructs the activation context. +* Test the proxy can handle large payloads (more than 1MB). +* Test the proxy does not permit re-initialization. +* Test the error handling for an action returning an invalid response. + +The canonical test suite should be extended by the new runtime tests. Additional +tests will be required depending on the feature set provided by the runtime. + +Since the OpenWhisk platform is language and runtime agnostic, it is generally not +necessary to add integration tests. That is the unit tests verifying the protocol are +sufficient. However, it may be necessary in some cases to modify the `wsk` CLI or +other OpenWhisk clients. In which case, appropriate tests should be added as necessary. +The OpenWhisk platform will perform a generic integration test as part of its basic +system tests. This integration test will require a [test function](#the-test-action) to +be available so that the test harness can create, invoke, and delete the action. diff --git a/docs/actions-swift.md b/docs/actions-swift.md index 859583d18a..5db368737b 100644 --- a/docs/actions-swift.md +++ b/docs/actions-swift.md @@ -100,9 +100,9 @@ wsk action invoke --result helloSwift --param name World ``` ```json - { - "greeting": "Hello World!" - } +{ + "greeting": "Hello World!" +} ``` Find out more about parameters in the [Working with parameters](./parameters.md) section. @@ -120,12 +120,12 @@ You can use a script to automate the packaging of the action. Create script `co set -ex if [ -z "$1" ] ; then - echo 'Error: Missing action name' - exit 1 + echo 'Error: Missing action name' + exit 1 fi if [ -z "$2" ] ; then - echo 'Error: Missing kind, for example swift:4.1' - exit 2 + echo 'Error: Missing kind, for example swift:4.1' + exit 2 fi OUTPUT_DIR="build" if [ ${2} == "swift:3.1.1" ]; then @@ -144,14 +144,14 @@ DEST_PACKAGE_SWIFT="$BASE_PATH/spm-build/Package.swift" BUILD_FLAGS="" if [ -n "$3" ] ; then - BUILD_FLAGS=${3} + BUILD_FLAGS=${3} fi echo "Using runtime $RUNTIME to compile swift" docker run --rm --name=compile-ow-swift -it -v "$(pwd):/owexec" $RUNTIME bash -ex -c " if [ -f \"/owexec/$OUTPUT_DIR/$1.zip\" ] ; then - rm \"/owexec/$OUTPUT_DIR/$1.zip\" + rm \"/owexec/$OUTPUT_DIR/$1.zip\" fi echo 'Setting up build...' @@ -159,8 +159,8 @@ cp /owexec/actions/$1/Sources/*.swift $DEST_SOURCE/ # action file can be either {action name}.swift or main.swift if [ -f \"$DEST_SOURCE/$1.swift\" ] ; then - echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift' - mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift + echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift' + mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift fi # Add in the OW specific bits cat $BASE_PATH/epilogue.swift >> $DEST_SOURCE/main.swift @@ -202,14 +202,14 @@ For Swift 3 here is an example: import PackageDescription let package = Package( - name: "Action", - dependencies: [ - .Package(url: "https://github.com/apple/example-package-deckofplayingcards.git", majorVersion: 3), - .Package(url: "https://github.com/IBM-Swift/CCurl.git", "0.2.3"), - .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.7.10"), - .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "15.0.1"), - .Package(url: "https://github.com/watson-developer-cloud/swift-sdk.git", "0.16.0") - ] + name: "Action", + dependencies: [ + .Package(url: "https://github.com/apple/example-package-deckofplayingcards.git", majorVersion: 3), + .Package(url: "https://github.com/IBM-Swift/CCurl.git", "0.2.3"), + .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.7.10"), + .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "15.0.1"), + .Package(url: "https://github.com/watson-developer-cloud/swift-sdk.git", "0.16.0") + ] ) ``` For Swift 4 here is an example: @@ -220,20 +220,20 @@ For Swift 3 here is an example: let package = Package( name: "Action", products: [ - .executable( - name: "Action", - targets: ["Action"] - ) + .executable( + name: "Action", + targets: ["Action"] + ) ], dependencies: [ - .package(url: "https://github.com/apple/example-package-deckofplayingcards.git", .upToNextMajor(from: "3.0.0")) + .package(url: "https://github.com/apple/example-package-deckofplayingcards.git", .upToNextMajor(from: "3.0.0")) ], targets: [ - .target( - name: "Action", - dependencies: ["DeckOfPlayingCards"], - path: "." - ) + .target( + name: "Action", + dependencies: ["DeckOfPlayingCards"], + path: "." + ) ] ) ``` @@ -281,7 +281,7 @@ enum VendingMachineError: Error { } func main(param: Input, completion: (Output?, Error?) -> Void) -> Void { // Return real error - do{ + do { throw VendingMachineError.insufficientFunds(coinsNeeded: 5) } catch { completion(nil, error) diff --git a/docs/actions.md b/docs/actions.md index 0097b12208..f365ccda73 100644 --- a/docs/actions.md +++ b/docs/actions.md @@ -51,16 +51,21 @@ Longer tutorials that are specific to a language of your choice are listed below We recommend reading the basics in this document first, which are language agnostic, before getting deeper into a language-specific tutorial. If your preferred language isn't supported directly, you may find the [Docker](docker-actions.md) action or [native binary](docker-actions.md#creating-native-actions) -paths more suitable. Multiple actions may be composed together to create a longer processing pipeline called a -[sequence](#creating-action-sequences). A more advanced form of composition is described [here](conductors.md). +paths more suitable. Or, you can [create a new runtime](#actions-new.md). -* [JavaScript](actions-node.md) -* [Python](actions-python.md) +* [Go](actions-go.md) * [Java](actions-java.md) +* [JavaScript](actions-node.md) * [PHP](actions-php.md) +* [Python](actions-python.md) * [Swift](actions-swift.md) * [Docker and native binaries](actions-docker.md) -* [Go](actions-go.md) + +Multiple actions from different languages may be composed together to create a longer processing +pipeline called a [sequence](#creating-action-sequences). The polyglot nature of the composition is +powerful in that it affords you the ability to use the right language for the problem you're solving, +and separates the orchestration of the dataflow between functions from the choice of language. +A more advanced form of composition is described [here](conductors.md). ## The basics ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
