This is an automated email from the ASF dual-hosted git repository.
rabbah pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/openwhisk-runtime-python.git
The following commit(s) were added to refs/heads/master by this push:
new 71ffd83 Updates to readme and packages (#117)
71ffd83 is described below
commit 71ffd83533a1b13c0130dc056ddc32b1d516f01c
Author: Sagar Manohar <[email protected]>
AuthorDate: Tue Oct 5 11:00:59 2021 -0400
Updates to readme and packages (#117)
Co-authored-by: Sagar Manohar <[email protected]>
---
README.md | 340 +++------------------------------
core/python36AiAction/Dockerfile | 4 +-
core/python36AiAction/build.gradle | 7 +
core/python36AiAction/requirements.txt | 54 ++----
core/python39Action/Dockerfile | 6 +-
core/python39Action/build.gradle | 7 +
core/python39Action/requirements.txt | 14 +-
core/python3Action/Dockerfile | 6 +-
core/python3Action/build.gradle | 12 ++
core/python3Action/requirements.txt | 13 +-
core/requirements_common.txt | 12 ++
README.md => tutorials/local_build.md | 182 +++---------------
12 files changed, 123 insertions(+), 534 deletions(-)
diff --git a/README.md b/README.md
index def82e3..021bc8a 100644
--- a/README.md
+++ b/README.md
@@ -17,331 +17,46 @@
#
-->
-# Apache OpenWhisk runtimes for Python
+# Apache OpenWhisk Runtimes for Python
[](http://www.apache.org/licenses/LICENSE-2.0)
[](https://travis-ci.com/apache/openwhisk-runtime-python)
-## Build Runtimes
-
-### You have 2 options to build the Python runtime:
-- Building locally
-- Using OpenWhisk Actions.
-### This README walks you through how to do both
-
-# Building Python Runtime Locally
-
-### Pre-requisites
-- [Docker](https://www.docker.com/)
-- [curl](https://curl.se/), [wget](https://www.gnu.org/software/wget/), or
[Postman](https://www.postman.com/)
-
-0. Choose/create a folder of your liking
-1. Clone this repo:
-```
-git clone https://github.com/apache/openwhisk-runtime-python
-cd openwhisk-runtime-python
-```
-
-2. Build docker
-
-Build using Python 3.7 (recommended)
-```
-docker build -t actionloop-python-v3.7:1.0-SNAPSHOT
$(pwd)/core/python3ActionLoop
-```
-
-For runtime 3.9 or 3.6-ai you need also to copy `bin` and `lib` folders from
3.7 in the Docker folder.
-
-
-2.1. Check docker `IMAGE ID` (3rd column) for repository
`actionloop-python-v3.7`
-```
-docker images
-```
-You should see an image that looks something like:
-```
-actionloop-python-v3.7 1.0-SNAPSHOT ...
-```
-
-2.2. Tag image (Optional step). Required if you’re pushing your docker image
to a registry e.g. dockerHub
-```
-docker tag <docker_image_ID>
<dockerHub_username>/actionloop-python-v3.7:1.0-SNAPSHOT
-```
-
-3. Run docker on localhost with either the following commands:
-```
-docker run -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
actionloop-python-v3.7:1.0-SNAPSHOT
-```
-Or run the container in the background (Add -d (detached) to the command above)
-```
-docker run -d -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
actionloop-python-v3.7:1.0-SNAPSHOT
-```
-Note: If you run your docker container in the background you'll want to stop
it with:
-```
-docker stop <container_id>
-```
-Where `<container_id>` is obtained from `docker ps` command bellow
-
-Lists all running containers
-```
-docker ps
-```
-or
-```
-docker ps -a
-```
-You shoulkd see a container named `bloom_whisker` being run
-
-4. Create your function (note that each container can only hold one function)
-In this first example we'll be creating a very simple function
-Create a json file called `python-data-init-run.json` which will contain the
function that looks something like the following:
-NOTE: value of code is the actual payload and must match the syntax of the
target runtime language, in this case `python`
-```json
-{
- "value": {
- "name" : "python-helloworld",
- "main" : "main",
- "binary" : false,
- "code" : "def main(args): return {'payload': 'Hello World!'}"
- }
-}
-```
-
-To issue the action against the running runtime, we must first make a request
against the `init` API
-We need to issue `POST` requests to init our function
-Using curl (the option `-d` signifies we're issuing a POST request)
-```
-curl -d "@python-data-init-run.json" -H "Content-Type: application/json"
http://localhost/init
-```
-Using wget (the option `--post-file` signifies we're issuing a POST request)
-```
-wget --post-file=python-data-init-run.json --header="Content-Type:
application/json" http://localhost/init
-```
-The above can also be achieved with [Postman](https://www.postman.com/) by
setting the headers and body accordingly
-
-Client expected response:
-```
-{"ok":true}
-```
-Server will remain silent in this case
-
-Now we can invoke/run our function against the `run` API with:
-Using curl `POST` request
-```
-curl -d "@python-data-init-run.json" -H "Content-Type: application/json"
http://localhost/run
-```
-Or using `GET` request
-```
-curl --data-binary "@python-data-init-run.json" -H "Content-Type:
application/json" http://localhost/run
-```
-Or
-Using wget `POST` request. The `-O-` is to redirect `wget` response to
`stdout`.
-```
-wget -O- --post-file=python-data-init-run.json --header="Content-Type:
application/json" http://localhost/run
-```
-Or using `GET` request
-```
-wget -O- --body-file=python-data-init-run.json --method=GET
--header="Content-Type: application/json" http://localhost/run
-```
-
-The above can also be achieved with [Postman](https://www.postman.com/) by
setting the headers and body accordingly.
-
-You noticed that we’re passing the same file `python-data-init-run.json` from
function initialization request to trigger the function. That’s not necessary
and not recommended since to trigger a function all we need is to pass the
parameters of the function. So in the above example, it's preferred if we
create a file called `python-data-params.json` that looks like the following:
-```json
-{
- "value": {}
-}
-```
-And trigger the function with the following (it also works with wget and
postman equivalents):
-```
-curl --data-binary "@python-data-params.json" -H "Content-Type:
application/json" http://localhost/run
-```
+This repository contains sources files needed to build the Python runtimes for
Apache OpenWhisk. The build system will produce a series of docker images for
each runtime version. These images are used in the platform to execute Python
actions.
-#### You can perform the same steps as above using
[Postman](https://www.postman.com/) application. Make sure you have the correct
request type set and the respective body. Also set the correct headers key
value pairs, which for us is "Content-Type: application/json"
+The following Python runtime versions (with kind & image labels) are generated
by the build system:
-After you trigger the function with one of the above commands you should
expect the following client response:
-```
-{"payload": "Hello World!"}
-```
-And Server expected response:
-```
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-```
-
-## Creating functions with arguments
-If your container still running from the previous example you must stop it and
re-run it before proceeding. Remember that each python runtime can only hold
one function (which cannot be overrided due to security reasons)
-Create a json file called `python-data-init-params.json` which will contain
the function to be initialized that looks like the following:
-```json
-{
- "value": {
- "name": "python-helloworld-with-params",
- "main" : "main",
- "binary" : false,
- "code" : "def main(args): return {'payload': 'Hello ' + args.get('name')
+ ' from ' + args.get('place') + '!!!'}"
- }
-}
-```
-Also create a json file `python-data-run-params.json` which will contain the
parameters to the function used to trigger it. Notice here we're creating 2
separate file from the beginning since this is good practice to make the
distinction between what needs to be sent via the `init` API and what needs to
be sent via the `run` API:
-```json
-{
- "value": {
- "name": "UFO",
- "place": "Mars"
- }
-}
-```
-
-Now, all we have to do is initialize and trigger our function.
-First, to initialize our function make sure your python runtime container is
running if not, spin the container by following step 3.
-Issue a `POST` request against the `init` API with the following command:
-Using curl:
-```
-curl -d "@python-data-init-params.json" -H "Content-Type: application/json"
http://localhost/init
-```
-Using wget:
-```
-wget --post-file=python-data-init-params.json --header="Content-Type:
application/json" http://localhost/init
-```
-Client expected response:
-```
-{"ok":true}
-```
-Server will remain silent in this case
-
-Second, to run/trigger the function issue requests against the `run` API with
the following command:
-Using curl with `POST`:
-```
-curl -d "@python-data-run-params.json" -H "Content-Type: application/json"
http://localhost/run
-```
-Or using curl with `GET`:
-```
-curl --data-binary "@python-data-run-params.json" -H "Content-Type:
application/json" http://localhost/run
-```
-Or
-Using wget with `POST`:
-```
-wget -O- --post-file=python-data-run-params.json --header="Content-Type:
application/json" http://localhost/run
-```
-Or using wget with `GET`:
-```
-wget -O- --body-file=python-data-run-params.json --method=GET
--header="Content-Type: application/json" http://localhost/run
-```
-
-After you trigger the function with one of the above commands you should
expect the following client response:
-```
-{"payload": "Hello UFO from Mars!!!"}
-```
-
-And Server expected response:
-```
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-```
+- Python 3.7 (python:3.7 & openwhisk/action-python-v3.7)
+- Python 3.9 (python:3.9 & openwhisk/action-python-v3.9)
+- Python 3.6 AI (python:3.6 & openwhisk/action-python-v3.6-ai)
-## Now let's create a more interesting function
-### This function will calculate the nth Fibonacci number
-This is the function we’re trying to create. It calculates the nth number of
the Fibonacci sequence recursively in `O(n)` time
-```python
-def fibonacci(n, mem):
- if (n == 0 or n == 1):
- return 1
- if (mem[n] == -1):
- mem[n] = fibonacci(n-1, mem) + fibonacci(n-2, mem)
- return mem[n]
-
-def main(args):
- n = int(args.get('fib_n'))
- mem = [-1 for i in range(n+1)]
- result = fibonacci(n, mem)
- key = 'Fibonacci of n == ' + str(n)
- return {key: result}
-```
+This README documents the build, customization and testing of these runtime
images.
-Create a json file called `python-fib-init.json` to initialize our fibonacci
function and insert the following. (It’s the same code as above but since we
can’t have a string span multiple lines in JSON we need to put all this code in
one line and this is how we do it. It’s ugly but not much we can do here)
-```json
-{
- "value": {
- "name": "python-recursive-fibonacci",
- "main" : "main",
- "binary" : false,
- "code" : "def fibonacci(n, mem):\n\tif (n == 0 or n == 1):\n\t\treturn
1\n\tif (mem[n] == -1):\n\t\tmem[n] = fibonacci(n-1, mem) + fibonacci(n-2,
mem)\n\treturn mem[n]\n\ndef main(args):\n\tn = int(args.get('fib_n'))\n\tmem =
[-1 for i in range(n+1)]\n\tresult = fibonacci(n, mem)\n\tkey = 'Fibonacci of n
== ' + str(n)\n\treturn {key: result}"
- }
-}
-```
-Create a json file called `python-fib-run.json` which will be used to
run/trigger our function with the appropriate argument:
-```json
-{
- "value": {
- "fib_n": "40"
- }
-}
-```
+To learn more about using Python actions to build serverless applications,
check out the main project documentation
[here](https://github.com/apache/openwhisk/blob/master/docs/actions-python.md).
-Now we’re all set.
-Make sure your python runtime container is running if not, spin the container
by following step 3.
-Initialize our fibonacci function by issuing a `POST` request against the
`init` API with the following command:
-Using curl:
-```
-curl -d "@python-fib-init.json" -H "Content-Type: application/json"
http://localhost/init
-```
-Using wget:
-```
-wget --post-file=python-fib-init.json --header="Content-Type:
application/json" http://localhost/init
-```
-Client expected response:
-```
-{"ok":true}
-```
-You've noticed by now that `init` API always returns `{"ok":true}` for a
successful initialized function. And the server, again, will remain silent
-
-Trigger the function by running/triggering the function with a request against
the `run` API with the following command:
-Using curl with `POST`:
-```
-curl -d "@python-fib-run.json" -H "Content-Type: application/json"
http://localhost/run
-```
-Using curl with `GET`:
-```
-curl --data-binary "@python-fib-run.json" -H "Content-Type: application/json"
http://localhost/run
-```
-Using wget with `POST`:
-```
-wget -O- --post-file=python-fib-run.json --header="Content-Type:
application/json" http://localhost/run
-```
-Using wget with `GET`:
-```
-wget -O- --body-file=python-fib-run.json --method=GET --header="Content-Type:
application/json" http://localhost/run
-```
-
-After you trigger the function with one of the above commands you should
expect the following client response:
-```
-{"Fibonacci of n == 40": 165580141}
-```
+## Build Runtimes
-And Server expected response:
-```
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
-```
+There are two options to build the Python runtime:
-#### At this point you can edit python-fib-run.json and try other `fib_n`
values. All you have to do is save `python-fib-run.json` and trigger the
function again. Notice that here we're just modifying the parameters of our
function; therefore, there's no need to re-run/re-initialize our container that
contains our Python runtime.
+- Building locally: [tutorial](tutorials/local_build.md)
+- Using OpenWhisk Actions.
-#### You can also automate most of this process through [docker
actions](https://github.com/apache/openwhisk/tree/master/tools/actionProxy) by
using `invoke.py`
+### Building Python Runtime using OpenWhisk Actions
-# Building Python Runtime using OpenWhisk Actions
+Pre-requisites
-### Pre-requisites
- [Gradle](https://gradle.org/)
- [Docker](https://www.docker.com/)
- [OpenWhisk CLI wsk](https://github.com/apache/openwhisk-cli/releases)
+The runtimes are built using Gradle. The file
[settings.gradle](settings.gradle) lists the images that are built by default.
-The runtimes are built using Gradle.
-The file [settings.gradle](settings.gradle) lists the images that are build by
default.
To build all those images, run the following command.
```
./gradlew distDocker
```
-You can optionally build a specific image by modifying the Gradle command. For
example:
+You can optionally build a specific image by modifying the gradle command. For
example:
```
./gradlew core:python3Action:distDocker
```
@@ -360,7 +75,7 @@ The Gradle build parameters `dockerImagePrefix` and
`dockerRegistry`
can be configured for your Docker Registry. Make sure you are logged
in first with the `docker` CLI.
-- Use the `docker` CLI to login. The following assume you will substitute
`$DOCKER_USER` with an appropriate value.
+- Use the `docker` CLI to login. The following assumes you will substitute
`$DOCKER_USER` with an appropriate value.
```
docker login --username $DOCKER_USER
```
@@ -403,25 +118,17 @@ others.
```
## Python 3 AI Runtime
-This action runtime enables developers to create AI Services with OpenWhisk.
It comes with preinstalled libraries useful for running machine learning and
deep learning inferences. [Read more about this runtime
here](./core/python3AiAction).
-
-## Import Project into IntelliJ
+This action runtime enables developers to create AI Services with OpenWhisk.
It comes with preinstalled libraries useful for running Machine Learning and
Deep Learning inferences. [Read more about this runtime
here](./core/python36AiAction).
-Follow these steps to import the project into your IntelliJ IDE.
-- Import project as gradle project.
-- Make sure the working directory is root of the project/repo.
-
-# Using extra libraries
+## Using additional python libraries
If you need more libraries for your Python action, you can include a
virtualenv in the zip file of the action.
The requirement is that the zip file must have a subfolder named `virtualenv`
with a script `virtualenv\bin\activate_this.py` working in an Linux AMD64
environment. It will be executed at start time to use your extra libraries.
-## Using requirements.txt
-
-Virtual envs are usually built listing your dependencies in a
`requirements.txt`.
+### Using requirements.txt
-If you have an action that requires addition libraries, you can just include
`requirements.txt`.
+Python virtual environments are typically built by installing dependencies
listed in a `requirements.txt` file. If you have an action that requires
additional libraries, you can include a `requirements.txt` file.
You have to create a folder `myaction` with at least two files:
@@ -434,8 +141,7 @@ Then zip your action and deploy to OpenWhisk, the
requirements will be installed
Keep in mind that resolving requirements involves downloading and install
software, so your action timeout limit may need to be adjusted accordingly.
Instead, you should consider using precompilation to resolve the requirements
at build time.
-
-## Precompilation of a virtualenv
+### Precompilation of a virtualenv
The action containers can actually generate a virtualenv for you, provided you
have a requirements.txt.
@@ -443,7 +149,7 @@ The action containers can actually generate a virtualenv
for you, provided you h
If you have an action in the format described before (with a
`requirements.txt`) you can build the zip file with the included files with:
```
-zip -j -r myaction | docker run -i action-python-v3.7 -compile main
>myaction.zip
+zip -j -r myaction | docker run -i action-python-v3.7 -compile main >
myaction.zip
```
You may use `v3.9` or `v3.6-ai` as well according to your Python version needs.
diff --git a/core/python36AiAction/Dockerfile b/core/python36AiAction/Dockerfile
index bb2dab6..434cb50 100644
--- a/core/python36AiAction/Dockerfile
+++ b/core/python36AiAction/Dockerfile
@@ -55,7 +55,7 @@ RUN apt-get update && apt-get upgrade -y && apt-get install
-y \
# PyTorch
# persistent as it fails often
RUN while ! pip list | grep torch ;\
- do pip install torch ; done ;\
+ do pip --no-cache-dir install torch ; done ;\
while ! pip list | grep torchvision ;\
do pip install torchvision ; done
@@ -64,6 +64,7 @@ RUN curl -L
https://downloads.rclone.org/rclone-current-linux-amd64.deb -o rclon
&& dpkg -i rclone.deb \
&& rm rclone.deb
+COPY requirements_common.txt requirements_common.txt
COPY requirements.txt requirements.txt
RUN pip3 install --upgrade pip six wheel &&\
pip3 install --no-cache-dir -r requirements.txt &&\
@@ -89,4 +90,5 @@ ENV OW_EXECUTION_ENV=openwhisk/action-python-v3.6-ai
ENV OW_COMPILER=/bin/compile
# use utf-8
ENV PYTHONIOENCODING=UTF-8
+
ENTRYPOINT ["/bin/proxy"]
diff --git a/core/python36AiAction/build.gradle
b/core/python36AiAction/build.gradle
index e915a75..c11c238 100644
--- a/core/python36AiAction/build.gradle
+++ b/core/python36AiAction/build.gradle
@@ -20,6 +20,7 @@ apply from: '../../gradle/docker.gradle'
distDocker.dependsOn 'copyLib'
distDocker.dependsOn 'copyBin'
+distDocker.dependsOn 'copyReqrCommon'
distDocker.finalizedBy('cleanup')
task copyLib(type: Copy) {
@@ -32,7 +33,13 @@ task copyBin(type: Copy) {
into './bin'
}
+task copyReqrCommon(type: Copy) {
+ from '../requirements_common.txt'
+ into './'
+}
+
task cleanup(type: Delete) {
delete 'bin'
delete 'lib'
+ delete 'requirements_common.txt'
}
diff --git a/core/python36AiAction/requirements.txt
b/core/python36AiAction/requirements.txt
index 44e5a9b..122027f 100644
--- a/core/python36AiAction/requirements.txt
+++ b/core/python36AiAction/requirements.txt
@@ -1,44 +1,32 @@
-# default available packages for python3action
-beautifulsoup4 == 4.8.2
-httplib2 == 0.17.0
-kafka_python == 1.4.7
-lxml == 4.5.0
-python-dateutil == 2.8.1
-requests == 2.22.0
-scrapy == 1.8.0
-simplejson == 3.17.0
-virtualenv == 16.7.9
-twisted == 19.10.0
-netifaces == 0.10.9
+# default packages available for action-python-v3.6-ai
+-r requirements_common.txt
# package to sync from a variety of cloud blob storage
python-rclone == 0.0.2
# more ML/DL packages
-keras == 2.3.1
-opencv-contrib-python == 4.2.0.32
-Cython == 0.29.14
-tools == 0.1.9
-scikit-image == 0.16.2
-
-nltk == 3.4.5
-
-PyJWT == 1.7.1
+keras==2.6.0
+opencv-contrib-python==4.5.3.56
+Cython==0.29.24
+tools==0.1.9
+scikit-image==0.17.2
+nltk==3.6.2
# packages for numerics
-numpy == 1.18.1
-scikit-learn == 0.22.1
-scipy == 1.4.1
-pandas == 1.0.1
+numpy==1.19.5
+scikit-learn==0.24.2
+scipy==1.5.4
+pandas==1.1.5
# packages for image processing
-Pillow == 7.0.0
+Pillow==8.3.2
# Etc
-pymongo == 3.10.1
-redis == 3.4.1
-pika == 1.1.0
-elasticsearch == 7.5.1
-cassandra-driver == 3.21.0
-etcd3 == 0.11.1
-twilio == 6.35.4
+PyJWT==1.7.1
+pymongo==3.12.0
+redis==3.5.3
+pika==1.2.0
+elasticsearch==7.14.1
+cassandra-driver==3.25.0
+etcd3==0.12.0
+twilio==6.63.2
diff --git a/core/python39Action/Dockerfile b/core/python39Action/Dockerfile
index 713a2ea..2366371 100644
--- a/core/python39Action/Dockerfile
+++ b/core/python39Action/Dockerfile
@@ -39,14 +39,18 @@ FROM python:3.9-buster
ARG GO_PROXY_BUILD_FROM=release
# Install common modules for python
+COPY requirements_common.txt requirements_common.txt
COPY requirements.txt requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
+RUN pip3 install --upgrade pip six wheel &&\
+ pip3 install --no-cache-dir -r requirements.txt
RUN mkdir -p /action
WORKDIR /
+
COPY --from=builder_source /bin/proxy /bin/proxy_source
COPY --from=builder_release /bin/proxy /bin/proxy_release
RUN mv /bin/proxy_${GO_PROXY_BUILD_FROM} /bin/proxy
+
ADD bin/compile /bin/compile
ADD lib/launcher.py /lib/launcher.py
diff --git a/core/python39Action/build.gradle b/core/python39Action/build.gradle
index 9abe626..3f46d75 100644
--- a/core/python39Action/build.gradle
+++ b/core/python39Action/build.gradle
@@ -20,6 +20,7 @@ apply from: '../../gradle/docker.gradle'
distDocker.dependsOn 'copyLib'
distDocker.dependsOn 'copyBin'
+distDocker.dependsOn 'copyReqrCommon'
distDocker.finalizedBy('cleanup')
task copyLib(type: Copy) {
@@ -32,7 +33,13 @@ task copyBin(type: Copy) {
into './bin'
}
+task copyReqrCommon(type: Copy) {
+ from '../requirements_common.txt'
+ into './'
+}
+
task cleanup(type: Delete) {
delete 'bin'
delete 'lib'
+ delete 'requirements_common.txt'
}
diff --git a/core/python39Action/requirements.txt
b/core/python39Action/requirements.txt
index 549a80f..377da9d 100644
--- a/core/python39Action/requirements.txt
+++ b/core/python39Action/requirements.txt
@@ -1,12 +1,2 @@
-# default available packages for action-python-v39
-beautifulsoup4 == 4.9.3
-httplib2 == 0.19.1
-kafka_python == 1.4.7
-lxml == 4.6.3
-python-dateutil == 2.8.1
-requests == 2.25.1
-scrapy == 1.8.0
-simplejson == 3.17.2
-virtualenv == 20.4.7
-twisted == 21.2.0
-netifaces == 0.11.0
+# default packages available for action-python-v3.9
+-r requirements_common.txt
diff --git a/core/python3Action/Dockerfile b/core/python3Action/Dockerfile
index ef4615a..9f00f24 100644
--- a/core/python3Action/Dockerfile
+++ b/core/python3Action/Dockerfile
@@ -39,14 +39,18 @@ FROM python:3.7-buster
ARG GO_PROXY_BUILD_FROM=release
# Install common modules for python
+COPY requirements_common.txt requirements_common.txt
COPY requirements.txt requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
+RUN pip3 install --upgrade pip six wheel &&\
+ pip3 install --no-cache-dir -r requirements.txt
RUN mkdir -p /action
WORKDIR /
+
COPY --from=builder_source /bin/proxy /bin/proxy_source
COPY --from=builder_release /bin/proxy /bin/proxy_release
RUN mv /bin/proxy_${GO_PROXY_BUILD_FROM} /bin/proxy
+
ADD bin/compile /bin/compile
ADD lib/launcher.py /lib/launcher.py
diff --git a/core/python3Action/build.gradle b/core/python3Action/build.gradle
index 527d982..e3b2382 100644
--- a/core/python3Action/build.gradle
+++ b/core/python3Action/build.gradle
@@ -17,3 +17,15 @@
ext.dockerImageName = 'action-python-v3.7'
apply from: '../../gradle/docker.gradle'
+
+distDocker.dependsOn 'copyReqrCommon'
+distDocker.finalizedBy('cleanup')
+
+task copyReqrCommon(type: Copy) {
+ from '../requirements_common.txt'
+ into './'
+}
+
+task cleanup(type: Delete) {
+ delete 'requirements_common.txt'
+}
diff --git a/core/python3Action/requirements.txt
b/core/python3Action/requirements.txt
index d60e88a..0ab215a 100644
--- a/core/python3Action/requirements.txt
+++ b/core/python3Action/requirements.txt
@@ -1,11 +1,2 @@
-# default available packages for python3action
-beautifulsoup4 == 4.6.3
-httplib2 == 0.11.3
-kafka_python == 1.4.3
-lxml == 4.2.5
-python-dateutil == 2.7.3
-requests == 2.19.1
-scrapy == 1.5.1
-simplejson == 3.16.0
-virtualenv == 16.0.0
-twisted == 18.7.0
+# default packages available for action-python-v3.7
+-r requirements_common.txt
diff --git a/core/requirements_common.txt b/core/requirements_common.txt
new file mode 100644
index 0000000..88d2c86
--- /dev/null
+++ b/core/requirements_common.txt
@@ -0,0 +1,12 @@
+# default packages available across all runtimes
+beautifulsoup4==4.10.0
+httplib2==0.19.1
+kafka_python==2.0.2
+lxml==4.6.3
+python-dateutil==2.8.2
+requests==2.26.0
+scrapy==2.5.0
+simplejson==3.17.5
+virtualenv==20.8.0
+twisted==21.7.0
+netifaces==0.11.0
diff --git a/README.md b/tutorials/local_build.md
similarity index 58%
copy from README.md
copy to tutorials/local_build.md
index def82e3..d1ffdaa 100644
--- a/README.md
+++ b/tutorials/local_build.md
@@ -17,20 +17,9 @@
#
-->
-# Apache OpenWhisk runtimes for Python
-[](http://www.apache.org/licenses/LICENSE-2.0)
-[](https://travis-ci.com/apache/openwhisk-runtime-python)
+# Building Python runtime locally
-## Build Runtimes
-
-### You have 2 options to build the Python runtime:
-- Building locally
-- Using OpenWhisk Actions.
-### This README walks you through how to do both
-
-# Building Python Runtime Locally
-
-### Pre-requisites
+## Pre-requisites
- [Docker](https://www.docker.com/)
- [curl](https://curl.se/), [wget](https://www.gnu.org/software/wget/), or
[Postman](https://www.postman.com/)
@@ -43,35 +32,33 @@ cd openwhisk-runtime-python
2. Build docker
-Build using Python 3.7 (recommended)
+Build using Python 3.7 (recommended). This tutorial assumes you're building
with python 3.7.
+
```
-docker build -t actionloop-python-v3.7:1.0-SNAPSHOT
$(pwd)/core/python3ActionLoop
+docker build -t action-python-v3.7:1.0-SNAPSHOT $(pwd)/core/python3Action
```
-For runtime 3.9 or 3.6-ai you need also to copy `bin` and `lib` folders from
3.7 in the Docker folder.
-
-
-2.1. Check docker `IMAGE ID` (3rd column) for repository
`actionloop-python-v3.7`
+2.1. Check docker `IMAGE ID` (3rd column) for repository `action-python-v3.7`
```
docker images
```
You should see an image that looks something like:
```
-actionloop-python-v3.7 1.0-SNAPSHOT ...
+action-python-v3.7 1.0-SNAPSHOT ...
```
2.2. Tag image (Optional step). Required if you’re pushing your docker image
to a registry e.g. dockerHub
```
-docker tag <docker_image_ID>
<dockerHub_username>/actionloop-python-v3.7:1.0-SNAPSHOT
+docker tag <docker_image_ID>
<dockerHub_username>/action-python-v3.7:1.0-SNAPSHOT
```
3. Run docker on localhost with either the following commands:
```
-docker run -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
actionloop-python-v3.7:1.0-SNAPSHOT
+docker run -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
action-python-v3.7:1.0-SNAPSHOT
```
Or run the container in the background (Add -d (detached) to the command above)
```
-docker run -d -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
actionloop-python-v3.7:1.0-SNAPSHOT
+docker run -d -p 127.0.0.1:80:8080/tcp --name=bloom_whisker --rm -it
action-python-v3.7:1.0-SNAPSHOT
```
Note: If you run your docker container in the background you'll want to stop
it with:
```
@@ -87,7 +74,7 @@ or
```
docker ps -a
```
-You shoulkd see a container named `bloom_whisker` being run
+You should see a container named `bloom_whisker` being run.
4. Create your function (note that each container can only hold one function)
In this first example we'll be creating a very simple function
@@ -122,7 +109,7 @@ Client expected response:
```
Server will remain silent in this case
-Now we can invoke/run our function against the `run` API with:
+Now we can invoke/run our function agains the `run` API with:
Using curl `POST` request
```
curl -d "@python-data-init-run.json" -H "Content-Type: application/json"
http://localhost/run
@@ -143,7 +130,7 @@ wget -O- --body-file=python-data-init-run.json --method=GET
--header="Content-Ty
The above can also be achieved with [Postman](https://www.postman.com/) by
setting the headers and body accordingly.
-You noticed that we’re passing the same file `python-data-init-run.json` from
function initialization request to trigger the function. That’s not necessary
and not recommended since to trigger a function all we need is to pass the
parameters of the function. So in the above example, it's preferred if we
create a file called `python-data-params.json` that looks like the following:
+You noticed that we’re passing the same file `python-data-init-run.json` from
function initialization request to trigger the function. That’s not necessary
and not recommended since to trigger a function all we need is to pass the
parameters of the function. So in the above example, it's prefered if we create
a file called `python-data-params.json` that looks like the following:
```json
{
"value": {}
@@ -154,7 +141,7 @@ And trigger the function with the following (it also works
with wget and postman
curl --data-binary "@python-data-params.json" -H "Content-Type:
application/json" http://localhost/run
```
-#### You can perform the same steps as above using
[Postman](https://www.postman.com/) application. Make sure you have the correct
request type set and the respective body. Also set the correct headers key
value pairs, which for us is "Content-Type: application/json"
+You can perform the same steps as above using
[Postman](https://www.postman.com/) application. Make sure you have the correct
request type set and the respective body. Also set the correct headers key
value pairs, which for us is "Content-Type: application/json"
After you trigger the function with one of the above commands you should
expect the following client response:
```
@@ -167,7 +154,8 @@ XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
```
## Creating functions with arguments
-If your container still running from the previous example you must stop it and
re-run it before proceeding. Remember that each python runtime can only hold
one function (which cannot be overrided due to security reasons)
+
+If your container still running from the previuous example you must stop it
and re-run it before proceding. Remember that each python runtime can only hold
one function (which cannot be overrided due to security reasons)
Create a json file called `python-data-init-params.json` which will contain
the function to be initialized that looks like the following:
```json
{
@@ -179,7 +167,7 @@ Create a json file called `python-data-init-params.json`
which will contain the
}
}
```
-Also create a json file `python-data-run-params.json` which will contain the
parameters to the function used to trigger it. Notice here we're creating 2
separate file from the beginning since this is good practice to make the
distinction between what needs to be sent via the `init` API and what needs to
be sent via the `run` API:
+Also create a json file `python-data-run-params.json` which will contain the
parameters to the function used to trigger it. Notice here we're creating 2
separate file from the beginning since this is good practice to make the
disticntion between what needs to be send via the `init` API and what needs to
be sent via the `run` API:
```json
{
"value": {
@@ -236,9 +224,10 @@ XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
```
-## Now let's create a more interesting function
-### This function will calculate the nth Fibonacci number
-This is the function we’re trying to create. It calculates the nth number of
the Fibonacci sequence recursively in `O(n)` time
+### Advanced function
+
+This function will calculate the nth Fibonacci number. It calculates the nth
number of the Fibonacci sequence recursively in `O(n)` time.
+
```python
def fibonacci(n, mem):
if (n == 0 or n == 1):
@@ -321,131 +310,8 @@ XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
```
-#### At this point you can edit python-fib-run.json and try other `fib_n`
values. All you have to do is save `python-fib-run.json` and trigger the
function again. Notice that here we're just modifying the parameters of our
function; therefore, there's no need to re-run/re-initialize our container that
contains our Python runtime.
-
-#### You can also automate most of this process through [docker
actions](https://github.com/apache/openwhisk/tree/master/tools/actionProxy) by
using `invoke.py`
-
-# Building Python Runtime using OpenWhisk Actions
-
-### Pre-requisites
-- [Gradle](https://gradle.org/)
-- [Docker](https://www.docker.com/)
-- [OpenWhisk CLI wsk](https://github.com/apache/openwhisk-cli/releases)
-
-
-The runtimes are built using Gradle.
-The file [settings.gradle](settings.gradle) lists the images that are build by
default.
-To build all those images, run the following command.
-
-```
-./gradlew distDocker
-```
-
-You can optionally build a specific image by modifying the Gradle command. For
example:
-```
-./gradlew core:python3Action:distDocker
-```
-
-The build will produce Docker images such as `action-python-v3.7`
-and will also tag the same image with the `whisk/` prefix. The latter
-is a convenience, which if you're testing with a local OpenWhisk
-stack, allows you to skip pushing the image to Docker Hub.
-
-The image will need to be pushed to Docker Hub if you want to test it
-with a hosted OpenWhisk installation.
-
-### Using Gradle to push to a Docker Registry
-
-The Gradle build parameters `dockerImagePrefix` and `dockerRegistry`
-can be configured for your Docker Registry. Make sure you are logged
-in first with the `docker` CLI.
-
-- Use the `docker` CLI to login. The following assume you will substitute
`$DOCKER_USER` with an appropriate value.
- ```
- docker login --username $DOCKER_USER
- ```
-
-- Now build, tag and push the image accordingly.
- ```
- ./gradlew distDocker -PdockerImagePrefix=$DOCKER_USER
-PdockerRegistry=docker.io
- ```
-
-### Using Your Image as an OpenWhisk Action
-
-You can now use this image as an OpenWhisk action. For example, to use
-the image `action-python-v3.7` as an action runtime, you would run
-the following command.
-
-```
-wsk action update myAction myAction.py --docker $DOCKER_USER/action-python-v3.7
-```
-
-## Test Runtimes
-
-There are suites of tests that are generic for all runtimes, and some that are
specific to a runtime version.
-To run all tests, there are two steps.
-
-First, you need to create an OpenWhisk snapshot release. Do this from your
OpenWhisk home directory.
-```
-./gradlew install
-```
-
-Now you can build and run the tests in this repository.
-```
-./gradlew tests:test
-```
-
-Gradle allows you to selectively run tests. For example, the following
-command runs tests which match the given pattern and excludes all
-others.
-```
-./gradlew :tests:test --tests Python*Tests
-```
-
-## Python 3 AI Runtime
-This action runtime enables developers to create AI Services with OpenWhisk.
It comes with preinstalled libraries useful for running machine learning and
deep learning inferences. [Read more about this runtime
here](./core/python3AiAction).
-
-## Import Project into IntelliJ
-
-Follow these steps to import the project into your IntelliJ IDE.
-- Import project as gradle project.
-- Make sure the working directory is root of the project/repo.
-
-# Using extra libraries
-
-If you need more libraries for your Python action, you can include a
virtualenv in the zip file of the action.
-
-The requirement is that the zip file must have a subfolder named `virtualenv`
with a script `virtualenv\bin\activate_this.py` working in an Linux AMD64
environment. It will be executed at start time to use your extra libraries.
-
-## Using requirements.txt
-
-Virtual envs are usually built listing your dependencies in a
`requirements.txt`.
-
-If you have an action that requires addition libraries, you can just include
`requirements.txt`.
-
-You have to create a folder `myaction` with at least two files:
-
-```
-__main__.py
-requirements.txt
-```
-
-Then zip your action and deploy to OpenWhisk, the requirements will be
installed for you at init time, creating a suitable virtualenv.
-
-Keep in mind that resolving requirements involves downloading and install
software, so your action timeout limit may need to be adjusted accordingly.
Instead, you should consider using precompilation to resolve the requirements
at build time.
-
-
-## Precompilation of a virtualenv
-
-The action containers can actually generate a virtualenv for you, provided you
have a requirements.txt.
-
-
-If you have an action in the format described before (with a
`requirements.txt`) you can build the zip file with the included files with:
-
-```
-zip -j -r myaction | docker run -i action-python-v3.7 -compile main
>myaction.zip
-```
+### Notes
-You may use `v3.9` or `v3.6-ai` as well according to your Python version needs.
+- At this point you can edit `python-fib-run.json` and try other `fib_n`
values. All you have to do is save `python-fib-run.json` and trigger the
function again. Notice that here we're just modifying the parameters of our
function; therefore, there's no need to re-run/re-initialize our container that
contains our Python runtime.
-The resulting action includes a virtualenv already built for you and that is
fast to deploy and start as all the dependencies are already resolved. Note
that there is a limit on the size of the zip file and this approach will not
work for installing large libraries like Pandas or Numpy, instead use the
provide "v.3.6-ai" runtime instead which provides these libraries already for
you.
+- You can also automate most of this process through [docker
actions](https://github.com/apache/openwhisk/tree/master/tools/actionProxy) by
using `invoke.py`
