[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672535#comment-16672535 ] ASF GitHub Bot commented on PIO-183: marevol closed pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docker/.ivy2/.keep b/docker/.ivy2/.keep new file mode 100644 index 0..ec2014340 --- /dev/null +++ b/docker/.ivy2/.keep @@ -0,0 +1,14 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/docker/JUPYTER.md b/docker/JUPYTER.md new file mode 100644 index 0..7a6e988aa --- /dev/null +++ b/docker/JUPYTER.md @@ -0,0 +1,166 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It helps you with your exploratory data analysis (EDA). + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/apache/predictionio-template-recommender.git +cd predictionio-template-recommender/ +``` + +Replace a name with `MyApp1`. + +``` +sed -i "s/INVALID_APP_NAME/MyApp1/" engine.json +``` + +### Register New Application + +Using pio command, register a new application as `MyApp1`. + +``` +pio app new MyApp1 +``` + +This command prints an access key as below. + +``` +[INFO] [Pio$] Access Key: bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +Set it to an environment variable `ACCESS_KEY`. + +``` +ACCESS_KEY=bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### Create Model + +To create a model, run: + +``` +pio train +``` + +## Getting Started With Python Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/jpioug/predictionio-template-iris.git +predictionio-template-iris/ +``` + +### Register New Application + +Using pio command, register a new application as `IrisApp`. + +``` +pio app new --access-key IRIS_TOKEN IrisApp +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +python data/import_eventserver.py +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### EDA + +To do data analysis, open `templates/predictionio-template-iris/eda.ipynb` on Jupyter. + +### Create Model + +You need to clear the following environment variables in the terminal before executing `pio train`. + +``` +unset PYSPARK_PYTHON +unset PYSPARK_DRIVER_PYTHON +unset PYSPARK_DRIVER_PYTHON_OPTS +``` + +To create a model, run: + +``` +pio train --main-py-file train.py +``` + + diff --git a/docker/README.md b/docker/README.md index f1bc5a08f..97e1bf59e 100644 --- a/docker/README.md +++
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661690#comment-16661690 ] ASF GitHub Bot commented on PIO-183: marevol commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227634435 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/apache/predictionio-template-recommender.git +cd predictionio-template-recommender/ +``` + +Replace a name with `MyApp1`. + +``` +sed -i "s/INVALID_APP_NAME/MyApp1/" engine.json +``` + +### Register New Application + +Using pio command, register a new application as `MyApp1`. + +``` +pio app new MyApp1 +``` + +This command prints an access key as below. + +``` +[INFO] [Pio$] Access Key: bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +Set it to an environment variable `ACCESS_KEY`. + +``` +ACCESS_KEY=bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### Create Model + +To create a model, run: + +``` +pio train +``` + +## Getting Started With Python Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/jpioug/predictionio-template-iris.git +predictionio-template-iris/ +``` + +### Register New Application + +Using pio command, register a new application as `IrisApp`. + +``` +pio app new --access-key IRIS_TOKEN IrisApp +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +python data/import_eventserver.py +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### EDA + +To do data analysis, open `templates/predictionio-template-iris/eda.ipynb` on Jupyter. + +### Create Model + +You need to clear the following environment variables in the terminal before executing `pio train`. + +``` +unset PYSPARK_PYTHON +unset PYSPARK_DRIVER_PYTHON +unset PYSPARK_DRIVER_PYTHON_OPTS +``` + +To create a model, run: + +``` +pio train --main-py-file train.py +``` Review comment: We can set settings other than Jupyter ones to PYSPARK_*. I think that it's better not to clear them in pio command... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661677#comment-16661677 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227632554 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/apache/predictionio-template-recommender.git +cd predictionio-template-recommender/ +``` + +Replace a name with `MyApp1`. + +``` +sed -i "s/INVALID_APP_NAME/MyApp1/" engine.json +``` + +### Register New Application + +Using pio command, register a new application as `MyApp1`. + +``` +pio app new MyApp1 +``` + +This command prints an access key as below. + +``` +[INFO] [Pio$] Access Key: bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +Set it to an environment variable `ACCESS_KEY`. + +``` +ACCESS_KEY=bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### Create Model + +To create a model, run: + +``` +pio train +``` + +## Getting Started With Python Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/jpioug/predictionio-template-iris.git +predictionio-template-iris/ +``` + +### Register New Application + +Using pio command, register a new application as `IrisApp`. + +``` +pio app new --access-key IRIS_TOKEN IrisApp +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +python data/import_eventserver.py +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### EDA + +To do data analysis, open `templates/predictionio-template-iris/eda.ipynb` on Jupyter. + +### Create Model + +You need to clear the following environment variables in the terminal before executing `pio train`. + +``` +unset PYSPARK_PYTHON +unset PYSPARK_DRIVER_PYTHON +unset PYSPARK_DRIVER_PYTHON_OPTS +``` + +To create a model, run: + +``` +pio train --main-py-file train.py +``` Review comment: Exactly, so I have come up with a following plan: if the `main-py-file` option is specified, PYSPARK_* environment variables are removed automatically before running `spark-submit` internally. Each has its pros and cons. It doesn't have to be implemented :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661620#comment-16661620 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227623514 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template Review comment: Wow, that's great! I'll give it a try. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660273#comment-16660273 ] ASF GitHub Bot commented on PIO-183: marevol commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227282935 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/apache/predictionio-template-recommender.git +cd predictionio-template-recommender/ +``` + +Replace a name with `MyApp1`. + +``` +sed -i "s/INVALID_APP_NAME/MyApp1/" engine.json +``` + +### Register New Application + +Using pio command, register a new application as `MyApp1`. + +``` +pio app new MyApp1 +``` + +This command prints an access key as below. + +``` +[INFO] [Pio$] Access Key: bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +Set it to an environment variable `ACCESS_KEY`. + +``` +ACCESS_KEY=bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### Create Model + +To create a model, run: + +``` +pio train +``` + +## Getting Started With Python Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/jpioug/predictionio-template-iris.git +predictionio-template-iris/ +``` + +### Register New Application + +Using pio command, register a new application as `IrisApp`. + +``` +pio app new --access-key IRIS_TOKEN IrisApp +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +python data/import_eventserver.py +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### EDA + +To do data analysis, open `templates/predictionio-template-iris/eda.ipynb` on Jupyter. + +### Create Model + +You need to clear the following environment variables in the terminal before executing `pio train`. + +``` +unset PYSPARK_PYTHON +unset PYSPARK_DRIVER_PYTHON +unset PYSPARK_DRIVER_PYTHON_OPTS +``` + +To create a model, run: + +``` +pio train --main-py-file train.py +``` Review comment: This Jupyter has PYSPARK_* environment variables. If these variables are not removed, `pio train` uses them and starts Jupyter in Jupyter... So, we need to clear them before executing `pio train` in Jupyter terminal. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660250#comment-16660250 ] ASF GitHub Bot commented on PIO-183: marevol commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227275805 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template Review comment: Yes. For scale template, you can run pio command and build it in Jupyter's terminal. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660252#comment-16660252 ] ASF GitHub Bot commented on PIO-183: marevol commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227275968 ## File path: docker/jupyter/start.sh ## @@ -0,0 +1,95 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +# store PIO environment to pio-env.sh +PIO_ENV_FILE=/etc/predictionio/pio-env.sh +env | grep ^PIO_ >> $PIO_ENV_FILE +if [ $(grep _MYSQL_ $PIO_ENV_FILE | wc -l) = 0 ] ; then + sed -i "s/^MYSQL/#MYSQL/" $PIO_ENV_FILE +fi + +# start event server +sh /usr/bin/pio_run & + +export PYSPARK_PYTHON=$CONDA_DIR/bin/python +export PYSPARK_DRIVER_PYTHON=$CONDA_DIR/bin/jupyter +export PYSPARK_DRIVER_PYTHON_OPTS=notebook Review comment: Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660051#comment-16660051 ] ASF GitHub Bot commented on PIO-183: marevol removed a comment on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431794678 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660053#comment-16660053 ] ASF GitHub Bot commented on PIO-183: marevol removed a comment on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431785932 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660052#comment-16660052 ] ASF GitHub Bot commented on PIO-183: marevol removed a comment on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431786116 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660050#comment-16660050 ] ASF GitHub Bot commented on PIO-183: marevol commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227217178 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660015#comment-16660015 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227203281 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template Review comment: This means that a user can use conventional templates in the notebook, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660016#comment-16660016 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227209093 ## File path: docker/jupyter/start.sh ## @@ -0,0 +1,95 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +# store PIO environment to pio-env.sh +PIO_ENV_FILE=/etc/predictionio/pio-env.sh +env | grep ^PIO_ >> $PIO_ENV_FILE +if [ $(grep _MYSQL_ $PIO_ENV_FILE | wc -l) = 0 ] ; then + sed -i "s/^MYSQL/#MYSQL/" $PIO_ENV_FILE +fi + +# start event server +sh /usr/bin/pio_run & + +export PYSPARK_PYTHON=$CONDA_DIR/bin/python +export PYSPARK_DRIVER_PYTHON=$CONDA_DIR/bin/jupyter +export PYSPARK_DRIVER_PYTHON_OPTS=notebook Review comment: There are duplicate processes in `start-jupyter.sh`. Are both necessary? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660018#comment-16660018 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227204283 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). Review comment: "It helps you with your exploratory data analysis (EDA)." is better. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660017#comment-16660017 ] ASF GitHub Bot commented on PIO-183: shimamoto commented on a change in pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#discussion_r227211667 ## File path: docker/JUPYTER.md ## @@ -0,0 +1,167 @@ + + +Jupyter With PredictionIO += + +## Overview + +Using Jupyter based docker, you can use Jupyter Notebook with PredictionIO environment. +It help your exploratory data analysis (EDA). + + +## Run Jupyter Notebook + +First of all, start Jupyter container with PredictionIO environment: + +``` +docker-compose -f docker-compose.jupyter.yml \ + -f pgsql/docker-compose.base.yml \ + -f pgsql/docker-compose.meta.yml \ + -f pgsql/docker-compose.event.yml \ + -f pgsql/docker-compose.model.yml \ + up +``` + +The above command prints a token to the console as below. + +``` +pio_1 | http://(3aaf67361022 or 127.0.0.1):/?token=e87a634b4ab7e2c8bcd86aea9def3eb48183c043eac86f3e +``` + +Open `http://127.0.0.1:/`, type the token, and then open a new terminal in Jupyter from `New` pulldown button. + +## Getting Started With Scala Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/apache/predictionio-template-recommender.git +cd predictionio-template-recommender/ +``` + +Replace a name with `MyApp1`. + +``` +sed -i "s/INVALID_APP_NAME/MyApp1/" engine.json +``` + +### Register New Application + +Using pio command, register a new application as `MyApp1`. + +``` +pio app new MyApp1 +``` + +This command prints an access key as below. + +``` +[INFO] [Pio$] Access Key: bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +Set it to an environment variable `ACCESS_KEY`. + +``` +ACCESS_KEY=bbe8xRHN1j3Sa8WeAT8TSxt5op3lUqhvXmKY1gLRjg70K-DUhHIJJ0-UzgKumxGm +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt +python data/import_eventserver.py --access_key $ACCESS_KEY +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### Create Model + +To create a model, run: + +``` +pio train +``` + +## Getting Started With Python Based Template + +### Download Template + +Clone a template using Git: + +``` +cd templates/ +git clone https://github.com/jpioug/predictionio-template-iris.git +predictionio-template-iris/ +``` + +### Register New Application + +Using pio command, register a new application as `IrisApp`. + +``` +pio app new --access-key IRIS_TOKEN IrisApp +``` + +### Import Training Data + +Download trainging data and import them to PredictionIO Event server. + +``` +python data/import_eventserver.py +``` + +### Build Template + +Build your template by the following command: + +``` +pio build --verbose +``` + +### EDA + +To do data analysis, open `templates/predictionio-template-iris/eda.ipynb` on Jupyter. + +### Create Model + +You need to clear the following environment variables in the terminal before executing `pio train`. + +``` +unset PYSPARK_PYTHON +unset PYSPARK_DRIVER_PYTHON +unset PYSPARK_DRIVER_PYTHON_OPTS +``` + +To create a model, run: + +``` +pio train --main-py-file train.py +``` Review comment: What do you think it is preferable to clear the above environment variables when running `pio train` in Python? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658868#comment-16658868 ] ASF GitHub Bot commented on PIO-183: marevol commented on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431786116 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658866#comment-16658866 ] ASF GitHub Bot commented on PIO-183: marevol commented on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431785932 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658853#comment-16658853 ] ASF GitHub Bot commented on PIO-183: marevol commented on issue #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481#issuecomment-431783078 I pushed docker images. You can try [JUPYTER.md](https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md) without building them. Please let me know if you have any questions/problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIO-183) Add Jupyter Docker image
[ https://issues.apache.org/jira/browse/PIO-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649598#comment-16649598 ] ASF GitHub Bot commented on PIO-183: marevol opened a new pull request #481: [PIO-183] Add Jupyter Docker image URL: https://github.com/apache/predictionio/pull/481 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Jupyter Docker image > > > Key: PIO-183 > URL: https://issues.apache.org/jira/browse/PIO-183 > Project: PredictionIO > Issue Type: New Feature >Reporter: Shinsuke Sugaya >Assignee: Shinsuke Sugaya >Priority: Major > > For usage, see > [JUPYTER.md|https://github.com/marevol/predictionio/blob/jupyter/docker/JUPYTER.md]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
