This is an automated email from the ASF dual-hosted git repository. shinsuke pushed a commit to branch develop in repository https://gitbox.apache.org/repos/asf/predictionio.git
The following commit(s) were added to refs/heads/develop by this push: new dc08628 [PIO-86] Docker support (#462) dc08628 is described below commit dc08628eb6489e594af369310bb1c9d36c81d15a Author: Shinsuke Sugaya <shinsuke.sug...@gmail.com> AuthorDate: Wed Oct 10 16:30:57 2018 +0900 [PIO-86] Docker support (#462) --- docker/README.md | 249 ++++++++++++++++++++++++++ docker/bin/pio-docker | 56 ++++++ docker/deploy/run.sh | 21 +++ docker/docker-compose.deploy.yml | 23 +++ docker/docker-compose.spark.yml | 34 ++++ docker/docker-compose.yml | 25 +++ docker/elasticsearch/docker-compose.base.yml | 41 +++++ docker/elasticsearch/docker-compose.event.yml | 22 +++ docker/elasticsearch/docker-compose.meta.yml | 22 +++ docker/localfs/docker-compose.model.yml | 27 +++ docker/mysql/docker-compose.base.yml | 34 ++++ docker/mysql/docker-compose.event.yml | 22 +++ docker/mysql/docker-compose.meta.yml | 22 +++ docker/mysql/docker-compose.model.yml | 22 +++ docker/pgsql/docker-compose.base.yml | 32 ++++ docker/pgsql/docker-compose.event.yml | 22 +++ docker/pgsql/docker-compose.meta.yml | 22 +++ docker/pgsql/docker-compose.model.yml | 22 +++ docker/pio/Dockerfile | 69 +++++++ docker/pio/pio_run | 64 +++++++ docker/templates/.keep | 14 ++ 21 files changed, 865 insertions(+) diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..f1bc5a0 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,249 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +Apache PredictionIO Docker +========================== + +## Overview + +PredictionIO Docker provides Docker image for use in development and production environment. + + +## Usage + +### Run PredictionIO with Selectable docker-compose Files + +You can choose storages for event/meta/model to select docker-compose.yml. + +``` +docker-compose -f docker-compose.yml -f ... up +``` + +Supported storages are as below: + +| Type | Storage | +|:-----:|:---------------------------------| +| Event | Postgresql, MySQL, Elasticsearch | +| Meta | Postgresql, MySQL, Elasticsearch | +| Model | Postgresql, MySQL, LocalFS | + +If you run PredictionIO with Postgresql, run as below: + +``` +docker-compose -f docker-compose.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +To use localfs as model storage, change as below: + +``` +docker-compose -f docker-compose.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f localfs/docker-compose.model.yml \ + up +``` + +## Tutorial + +In this demo, we will show you how to build a recommendation template. + +### Run PredictionIO environment + +The following command starts PredictionIO with an event server. +PredictionIO docker image mounts ./templates directory to /templates. + +``` +$ docker-compose -f docker-compose.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +We provide `pio-docker` command as an utility for `pio` command. +`pio-docker` invokes `pio` command in PredictionIO container. + +``` +$ export PATH=`pwd`/bin:$PATH +$ pio-docker status +... +[INFO] [Management$] Your system is all ready to go. +``` + +### Download Recommendation Template + +This demo uses [predictionio-template-recommender](https://github.com/apache/predictionio-template-recommender). + +``` +$ cd templates +$ git clone https://github.com/apache/predictionio-template-recommender.git MyRecommendation +$ cd MyRecommendation +``` + +### Register Application + +You need to register this application to PredictionIO: + +``` +$ pio-docker app new MyApp1 +[INFO] [App$] Initialized Event Store for this app ID: 1. +[INFO] [Pio$] Created a new app: +[INFO] [Pio$] Name: MyApp1 +[INFO] [Pio$] ID: 1 +[INFO] [Pio$] Access Key: i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl +``` + +Since an access key is required in subsequent steps, set it to ACCESS_KEY. + +``` +$ ACCESS_KEY=i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl +``` + +`engine.json` contains an application name, so replace `INVALID_APP_NAME` with `MyApp1`. + +``` +... +"datasource": { + "params" : { + "appName": "MyApp1" + } +}, +... +``` + +### Import Data + +To import training data to Event server for PredictionIO, this template provides an import tool. +The tool depends on PredictionIO Python SDK and install as below: + +``` +$ pip install predictionio +``` +and then import data: +``` +$ curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +$ python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +This is Scala based template. +So, you need to build this template by `pio` command. + +``` +$ pio-docker build --verbose +``` + +### Train and Create Model + +To train a recommendation model, run `train` sub-command: + +``` +$ pio-docker train +``` + +### Deploy Model + +If a recommendation model is created successfully, deploy it to Prediction server for PredictionIO. + +``` +$ pio-docker deploy + +``` +You can check predictions as below: +``` +$ curl -H "Content-Type: application/json" \ +-d '{ "user": "1", "num": 4 }' http://localhost:8000/queries.json +``` + +## Advanced Topics + +### Run with Elasticsearch + +For Elasticsearch, Meta and Event storage are available. +To start PredictionIO with Elasticsearch, + +``` +docker-compose -f docker-compose.yml \ + -f elasticsearch/docker-compose.base.yml \ + -f elasticsearch/docker-compose.meta.yml \ + -f elasticsearch/docker-compose.event.yml \ + -f localfs/docker-compose.model.yml \ + up +``` + +### Run with Spark Cluster + +Adding `docker-compose.spark.yml`, you can use Spark cluster on `pio train`. + +``` +docker-compose -f docker-compose.yml \ + -f docker-compose.spark.yml \ + -f elasticsearch/docker-compose.base.yml \ + -f elasticsearch/docker-compose.meta.yml \ + -f elasticsearch/docker-compose.event.yml \ + -f localfs/docker-compose.model.yml \ + up +``` + +To submit a training task to Spark Cluster, run `pio-deploy train` with `--master` option: + +``` +pio-docker train -- --master spark://spark-master:7077 +``` + +See `docker-compose.spark.yml` if changing settings for Spark Cluster. + +### Run Engine Server + +To deploy your engine and start an engine server, run Docker with `docker-compose.deploy.yml`. + +``` +docker-compose -f docker-compose.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + -f docker-compose.deploy.yml \ + up +``` + +See `deploy/run.sh` and `docker-compose.deploy.yml` if changing a deployment. + + +## Development + +### Build Docker Image + +``` +docker build -t predictionio/pio pio +``` + +### Push Docker Image + +``` +docker push predictionio/pio:latest +docker tag predictionio/pio:latest predictionio/pio:$PIO\_VERSION +docker push predictionio/pio:$PIO\_VERSION +``` diff --git a/docker/bin/pio-docker b/docker/bin/pio-docker new file mode 100755 index 0000000..8046060 --- /dev/null +++ b/docker/bin/pio-docker @@ -0,0 +1,56 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +BASE_WORK_DIR=/templates +CURRENT_DIR=`pwd` + +get_container_id() { + if [ x"$PIO_CONTAINER_ID" != "x" ] ; then + echo $PIO_CONTAINER_ID + return + fi + for i in `docker ps -f "name=pio" -q` ; do + echo $i + return + done +} + +get_current_dir() { + if [ x"$PIO_CURRENT_DIR" != "x" ] ; then + echo $PIO_CURRENT_DIR + return + fi + D=`echo $CURRENT_DIR | sed -e "s,.*$BASE_WORK_DIR,$BASE_WORK_DIR,"` + if [[ $D = $BASE_WORK_DIR* ]] ; then + echo $D + else + echo $BASE_WORK_DIR + fi +} + +cid=`get_container_id` +if [ x"$cid" = "x" ] ; then + echo "Docker Container is not found." + exit 1 +fi + +wdir=`get_current_dir` + +docker exec -w $wdir -it $cid pio $@ + diff --git a/docker/deploy/run.sh b/docker/deploy/run.sh new file mode 100644 index 0000000..f225a62 --- /dev/null +++ b/docker/deploy/run.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +cd /templates/$PIO_TEMPLATE_NAME +pio deploy diff --git a/docker/docker-compose.deploy.yml b/docker/docker-compose.deploy.yml new file mode 100644 index 0000000..145a960 --- /dev/null +++ b/docker/docker-compose.deploy.yml @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + - "PIO_TEMPLATE_NAME=MyRecommendation" + - "PIO_RUN_FILE=/deploy/run.sh" + volumes: + - ./deploy:/deploy diff --git a/docker/docker-compose.spark.yml b/docker/docker-compose.spark.yml new file mode 100644 index 0000000..8f3c43e --- /dev/null +++ b/docker/docker-compose.spark.yml @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + spark-master: + image: bde2020/spark-master:2.2.2-hadoop2.7 + container_name: spark-master + ports: + - "8080:8080" + - "7077:7077" + environment: + - INIT_DAEMON_STEP=setup_spark + spark-worker-1: + image: bde2020/spark-worker:2.2.2-hadoop2.7 + container_name: spark-worker-1 + depends_on: + - spark-master + ports: + - "8081:8081" + environment: + - "SPARK_MASTER=spark://spark-master:7077" diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml new file mode 100644 index 0000000..e42c907 --- /dev/null +++ b/docker/docker-compose.yml @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + image: predictionio/pio:latest + ports: + - 7070:7070 + - 8000:8000 + volumes: + - ./templates:/templates + dns: 8.8.8.8 diff --git a/docker/elasticsearch/docker-compose.base.yml b/docker/elasticsearch/docker-compose.base.yml new file mode 100644 index 0000000..4784f4a --- /dev/null +++ b/docker/elasticsearch/docker-compose.base.yml @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4 + environment: + - xpack.graph.enabled=false + - xpack.ml.enabled=false + - xpack.monitoring.enabled=false + - xpack.security.enabled=false + - xpack.watcher.enabled=false + - cluster.name=predictionio + - bootstrap.memory_lock=true + - "ES_JAVA_OPTS=-Xms1g -Xmx1g" + ulimits: + memlock: + soft: -1 + hard: -1 + pio: + depends_on: + - elasticsearch + environment: + PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE: elasticsearch + PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS: elasticsearch + PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS: 9200 + PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES: http + diff --git a/docker/elasticsearch/docker-compose.event.yml b/docker/elasticsearch/docker-compose.event.yml new file mode 100644 index 0000000..5a77b6a --- /dev/null +++ b/docker/elasticsearch/docker-compose.event.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event + PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: ELASTICSEARCH + diff --git a/docker/elasticsearch/docker-compose.meta.yml b/docker/elasticsearch/docker-compose.meta.yml new file mode 100644 index 0000000..0ce31dd --- /dev/null +++ b/docker/elasticsearch/docker-compose.meta.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta + PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: ELASTICSEARCH + diff --git a/docker/localfs/docker-compose.model.yml b/docker/localfs/docker-compose.model.yml new file mode 100644 index 0000000..f38ff53 --- /dev/null +++ b/docker/localfs/docker-compose.model.yml @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model + PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: LOCALFS + PIO_FS_BASEDIR: /work/pio_store + PIO_FS_ENGINESDIR: /work/pio_store/engines + PIO_FS_TMPDIR: /work/pio_store/tmp + PIO_STORAGE_SOURCES_LOCALFS_TYPE: localfs + PIO_STORAGE_SOURCES_LOCALFS_PATH: /work/pio_store/models + diff --git a/docker/mysql/docker-compose.base.yml b/docker/mysql/docker-compose.base.yml new file mode 100644 index 0000000..fad5309 --- /dev/null +++ b/docker/mysql/docker-compose.base.yml @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + mysql: + image: mysql:8 + command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci + environment: + MYSQL_ROOT_PASSWORD: root + MYSQL_USER: pio + MYSQL_PASSWORD: pio + MYSQL_DATABASE: pio + pio: + depends_on: + - mysql + environment: + PIO_STORAGE_SOURCES_MYSQL_TYPE: jdbc + PIO_STORAGE_SOURCES_MYSQL_URL: "jdbc:mysql://mysql/pio" + PIO_STORAGE_SOURCES_MYSQL_USERNAME: pio + PIO_STORAGE_SOURCES_MYSQL_PASSWORD: pio + diff --git a/docker/mysql/docker-compose.event.yml b/docker/mysql/docker-compose.event.yml new file mode 100644 index 0000000..f5f4035 --- /dev/null +++ b/docker/mysql/docker-compose.event.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event + PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: MYSQL + diff --git a/docker/mysql/docker-compose.meta.yml b/docker/mysql/docker-compose.meta.yml new file mode 100644 index 0000000..f7a5ece --- /dev/null +++ b/docker/mysql/docker-compose.meta.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta + PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: MYSQL + diff --git a/docker/mysql/docker-compose.model.yml b/docker/mysql/docker-compose.model.yml new file mode 100644 index 0000000..4a48684 --- /dev/null +++ b/docker/mysql/docker-compose.model.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model + PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: MYSQL + diff --git a/docker/pgsql/docker-compose.base.yml b/docker/pgsql/docker-compose.base.yml new file mode 100644 index 0000000..297d6a8 --- /dev/null +++ b/docker/pgsql/docker-compose.base.yml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + postgres: + image: postgres:9 + environment: + POSTGRES_USER: pio + POSTGRES_PASSWORD: pio + POSTGRES_INITDB_ARGS: --encoding=UTF8 + pio: + depends_on: + - postgres + environment: + PIO_STORAGE_SOURCES_PGSQL_TYPE: jdbc + PIO_STORAGE_SOURCES_PGSQL_URL: "jdbc:postgresql://postgres/pio" + PIO_STORAGE_SOURCES_PGSQL_USERNAME: pio + PIO_STORAGE_SOURCES_PGSQL_PASSWORD: pio + diff --git a/docker/pgsql/docker-compose.event.yml b/docker/pgsql/docker-compose.event.yml new file mode 100644 index 0000000..2579141 --- /dev/null +++ b/docker/pgsql/docker-compose.event.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event + PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: PGSQL + diff --git a/docker/pgsql/docker-compose.meta.yml b/docker/pgsql/docker-compose.meta.yml new file mode 100644 index 0000000..345bfbe --- /dev/null +++ b/docker/pgsql/docker-compose.meta.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta + PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: PGSQL + diff --git a/docker/pgsql/docker-compose.model.yml b/docker/pgsql/docker-compose.model.yml new file mode 100644 index 0000000..329649f --- /dev/null +++ b/docker/pgsql/docker-compose.model.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + pio: + environment: + PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model + PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: PGSQL + diff --git a/docker/pio/Dockerfile b/docker/pio/Dockerfile new file mode 100644 index 0000000..88930bd --- /dev/null +++ b/docker/pio/Dockerfile @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +FROM openjdk:8 + +ARG PIO_GIT_URL=https://github.com/apache/predictionio.git +ARG PIO_TAG=v0.13.0 +ENV SCALA_VERSION=2.11.12 +ENV SPARK_VERSION=2.2.2 +ENV HADOOP_VERSION=2.7.7 +ENV ELASTICSEARCH_VERSION=5.5.3 +ENV PGSQL_VERSION=42.2.4 +ENV MYSQL_VERSION=8.0.12 +ENV PIO_HOME=/usr/share/predictionio + +RUN apt-get update && \ + apt-get install -y dpkg-dev fakeroot && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +WORKDIR /opt/src +RUN git clone -b $PIO_TAG $PIO_GIT_URL +WORKDIR /opt/src/predictionio +RUN bash ./make-distribution.sh \ + -Dscala.version=$SCALA_VERSION \ + -Dspark.version=$SPARK_VERSION \ + -Dhadoop.version=$HADOOP_VERSION \ + -Delasticsearch.version=$ELASTICSEARCH_VERSION \ + --with-deb && \ + dpkg -i ./assembly/target/predictionio_*.deb && \ + rm -rf /opt/src/predictionio/* + + +RUN cp /etc/predictionio/pio-env.sh /etc/predictionio/pio-env.sh.orig && \ + echo "#!/usr/bin/env bash" > /etc/predictionio/pio-env.sh +RUN curl -o $PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar \ + http://central.maven.org/maven2/org/postgresql/postgresql/$PGSQL_VERSION/postgresql-$PGSQL_VERSION.jar && \ + echo "POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar" >> /etc/predictionio/pio-env.sh && \ + echo "MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar" >> /etc/predictionio/pio-env.sh + +WORKDIR /usr/share +RUN curl -o /opt/src/spark-$SPARK_VERSION.tgz \ + http://www-us.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz && \ + tar zxvf /opt/src/spark-$SPARK_VERSION.tgz && \ + echo "SPARK_HOME="`pwd`/`ls -d spark*` >> /etc/predictionio/pio-env.sh && \ + rm -rf /opt/src + +WORKDIR /templates +ADD pio_run /usr/bin/pio_run + +EXPOSE 7070 +EXPOSE 8000 + +CMD ["sh", "/usr/bin/pio_run"] + diff --git a/docker/pio/pio_run b/docker/pio/pio_run new file mode 100644 index 0000000..83ac6cc --- /dev/null +++ b/docker/pio/pio_run @@ -0,0 +1,64 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +. /etc/predictionio/pio-env.sh + +# check elasticsearch status +if [ x"$PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE" != "x" ] ; then + RET=-1 + COUNT=0 + ES_HOST=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS | sed -e "s/,.*//"` + ES_PORT=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS | sed -e "s/,.*//"` + # Wait for elasticsearch startup + while [ $RET != 0 -a $COUNT -lt 10 ] ; do + echo "Waiting for ${ES_HOST}..." + curl --connect-timeout 60 --retry 10 -s "$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=green&timeout=1m" + RET=$? + COUNT=`expr $COUNT + 1` + sleep 1 + done +fi + +# check mysql jar file +if [ x"$PIO_STORAGE_SOURCES_MYSQL_TYPE" != "x" ] ; then + MYSQL_JAR_FILE=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar + if [ ! -f $MYSQL_JAR_FILE ] ; then + curl -o $MYSQL_JAR_FILE http://central.maven.org/maven2/mysql/mysql-connector-java/$MYSQL_VERSION/mysql-connector-java-$MYSQL_VERSION.jar + fi +fi + +# Check PIO status +RET=-1 +COUNT=0 +while [ $RET != 0 -a $COUNT -lt 10 ] ; do + echo "Waiting for PredictionIO..." + $PIO_HOME/bin/pio status + RET=$? + COUNT=`expr $COUNT + 1` + sleep 1 +done + + +if [ x"$PIO_RUN_FILE" != "x" ] ; then + sh $PIO_RUN_FILE +else + # Start PIO Event Server + $PIO_HOME/bin/pio eventserver +fi + diff --git a/docker/templates/.keep b/docker/templates/.keep new file mode 100644 index 0000000..ec20143 --- /dev/null +++ b/docker/templates/.keep @@ -0,0 +1,14 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License.