incubator-predictionio-template-recommender
On Wed, Sep 27, 2017 at 2:25 PM, Donald Szeto <don...@apache.org> wrote: > What template and version are you using? > > On Tue, Sep 26, 2017 at 9:28 PM Jim Miller <jemiller1...@gmail.com> wrote: > >> Hi guys, >> >> I have another error that I cannot get past: >> >> *vagrant:MyRecommendation/ (develop*) $ pio train >> >> [21:47:39]* >> *[ERROR] [Console$] unable to read engine factory from >> /home/vagrant/MyRecommendation/engine.json. Aborting.* >> >> >> Does anyone have any advice? >> >> Thanks, >> >> Jim >> >> -- >> Jim Miller >> >> On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1...@gmail.com) >> wrote: >> >> Shinsuke, you rock! That was it. >> >> Thank you so much, >> >> Jim >> >> -- >> Jim Miller >> >> On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya ( >> shinsuke.sug...@gmail.com) wrote: >> >> Hi, >> >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >> >> Elasticsearch 5 uses 9200 port. >> Could you try 9200 port? >> >> Thanks, >> shinsuke >> >> >> 2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1...@gmail.com>: >> > Hi Donald, >> > >> > Tried just now and received the following error: >> > >> > vagrant:~/ $ pio status >> > [13:34:52] >> > [INFO] [Management$] Inspecting PredictionIO... >> > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at >> > /home/vagrant/pio/PredictionIO-0.12.0-incubating >> > [INFO] [Management$] Inspecting Apache Spark... >> > [INFO] [Management$] Apache Spark is installed at >> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1. >> 1-bin-hadoop2.7 >> > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum >> requirement >> > of 1.3.0) >> > [INFO] [Management$] Inspecting storage backend connections... >> > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)... >> > [ERROR] [Management$] Unable to connect to all storage backends >> > successfully. >> > The following shows the error message from the storage backend. >> > >> > error while performing request (java.lang.RuntimeException) >> > >> > Dumping configuration of initialized storage backend sources. >> > Please make sure they are correct. >> > >> > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> >> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/ >> elasticsearch-5.5.2, >> > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE -> >> > elastic search >> > >> > >> > HERE IS MY PIO-ENV.SH >> > >> > # PredictionIO Main Configuration >> > # >> > # This section controls core behavior of PredictionIO. It is very likely >> > that >> > # you need to change these to fit your site. >> > >> > # SPARK_HOME: Apache Spark is a hard dependency and must be configured. >> > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7 >> > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7 >> > >> > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar >> > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar >> > >> > # ES_CONF_DIR: You must configure this if you have advanced >> configuration >> > for >> > # your Elasticsearch setup. >> > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2 >> > # HADOOP_CONF_DIR=/opt/hadoop >> > >> > # HBASE_CONF_DIR: You must configure this if you intend to run >> PredictionIO >> > # with HBase on a remote cluster. >> > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf >> > >> > # Filesystem paths where PredictionIO uses as block storage. >> > PIO_FS_BASEDIR=$HOME/.pio_store >> > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines >> > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp >> > >> > # PredictionIO Storage Configuration >> > # >> > # This section controls programs that make use of PredictionIO's >> built-in >> > # storage facilities. Default values are shown below. >> > # >> > # For more information on storage configuration please refer to >> > # http://predictionio.incubator.apache.org/system/anotherdatastore/ >> > >> > # Storage Repositories >> > >> > # Default is to use PostgreSQL >> > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta >> > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH >> > >> > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event >> > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE >> > >> > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model >> > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS >> > >> > # Storage Data Sources >> > >> > # PostgreSQL Default Settings >> > # Please change "pio" to your database name in >> PIO_STORAGE_SOURCES_PGSQL_URL >> > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and >> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly >> > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc >> > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio >> > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio >> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio >> > >> > # MySQL Example >> > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc >> > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio >> > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio >> > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio >> > >> > # Elasticsearch Example >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200 >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http >> > # >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/ >> vendors/elasticsearch-5.5.2 >> > # Optional basic HTTP auth >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name >> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret >> > # Elasticsearch 1.x Example >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/ >> vendors/elasticsearch-5.5.2 >> > >> > # Local File System Example >> > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >> > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models >> > >> > # HBase Example >> > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase >> > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1 >> > >> > # AWS S3 Example >> > # PIO_STORAGE_SOURCES_S3_TYPE=s3 >> > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket >> > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model >> > >> > >> > ELASTICSEARCH.YML >> > >> > #======================== Elasticsearch Configuration >> > ========================= >> > # >> > # NOTE: Elasticsearch comes with reasonable defaults for most settings. >> > # Before you set out to tweak and tune the configuration, make sure >> > you >> > # understand what are you trying to accomplish and the consequences. >> > # >> > # The primary way of configuring a node is via this file. This template >> > lists >> > # the most important settings you may want to configure for a production >> > cluster. >> > # >> > # Please consult the documentation for further information on >> configuration >> > options: >> > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html >> > # >> > # ---------------------------------- Cluster >> > ----------------------------------- >> > # >> > # Use a descriptive name for your cluster: >> > # >> > cluster.name: firstCluster >> > # >> > # ------------------------------------ Node >> > ------------------------------------ >> > # >> > # Use a descriptive name for the node: >> > # >> > #node.name: node-1 >> > # >> > # Add custom attributes to the node: >> > # >> > #node.attr.rack: r1 >> > # >> > # ----------------------------------- Paths >> > ------------------------------------ >> > # >> > # Path to directory where to store the data (separate multiple >> locations by >> > comma): >> > # >> > #path.data: /path/to/data >> > # >> > # Path to log files: >> > # >> > #path.logs: /path/to/logs >> > # >> > # ----------------------------------- Memory >> > ----------------------------------- >> > # >> > # Lock the memory on startup: >> > # >> > #bootstrap.memory_lock: true >> > # >> > # Make sure that the heap size is set to about half the memory available >> > # on the system and that the owner of the process is allowed to use this >> > # limit. >> > # >> > # Elasticsearch performs poorly when the system is swapping the memory. >> > # >> > # ---------------------------------- Network >> > ----------------------------------- >> > # >> > # Set the bind address to a specific IP (IPv4 or IPv6): >> > # >> > #network.host: 192.168.0.1 >> > # >> > # Set a custom port for HTTP: >> > # >> > #http.port: 9200 >> > # >> > # For more information, consult the network module documentation. >> > # >> > # --------------------------------- Discovery >> > ---------------------------------- >> > # >> > # Pass an initial list of hosts to perform discovery when new node is >> > started: >> > # The default list of hosts is ["127.0.0.1", "[::1]"] >> > # >> > #discovery.zen.ping.unicast.hosts: ["host1", "host2"] >> > # >> > # Prevent the "split brain" by configuring the majority of nodes (total >> > number of master-eligible nodes / 2 + 1): >> > # >> > #discovery.zen.minimum_master_nodes: 3 >> > # >> > # For more information, consult the zen discovery module documentation. >> > # >> > # ---------------------------------- Gateway >> > ----------------------------------- >> > # >> > # Block initial recovery after a full cluster restart until N nodes are >> > started: >> > # >> > #gateway.recover_after_nodes: 3 >> > # >> > # For more information, consult the gateway module documentation. >> > # >> > # ---------------------------------- Various >> > ----------------------------------- >> > # >> > # Require explicit names when deleting indices: >> > # >> > #action.destructive_requires_name: true >> > >> > >> > Thanks for all of your help! >> > >> > Jim >> > -- >> > Jim Miller >> > >> > On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1...@gmail.com >> ) >> > wrote: >> > >> > Hi Donald, >> > >> > I did not. I will read the release notes and update accordingly. >> > >> > Thanks! >> > >> > Jim >> > >> > -- >> > Jim Miller >> > >> > On September 20, 2017 at 1:01:53 PM, Donald Szeto (don...@apache.org) >> wrote: >> > >> > Hey Jim, >> > >> > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated >> in >> > 0.12 so the default build will use ES 5.x support. >> > >> > See the upcoming release notes: >> > https://github.com/apache/incubator-predictionio/blob/ >> release/0.12.0/RELEASE.md#behavior-changes >> > >> > Regards, >> > Donald >> > >> > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1...@gmail.com> >> wrote: >> >> >> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0: >> >> >> >> https://medium.freecodecamp.org/building-an- >> recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8 >> >> >> >> -- >> >> Jim Miller >> >> >> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel ( >> p...@occamsmachete.com) >> >> wrote: >> >> >> >> meaning is “firstcluster” the cluster name in your Elasticsearch >> >> configuration? >> >> >> >> >> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan...@gmail.com> >> wrote: >> >> >> >> I think the problem is with Elasticsearch, are you sure the cluster >> exists >> >> in elasticsearch configuration? >> >> >> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1...@gmail.com> >> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase: >> >>> >> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4 >> >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0 >> >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6 >> >>> >> >>> >> >>> All starts with no errors but with pio status I get: >> >>> >> >>> [INFO] [Management$] Inspecting PredictionIO... >> >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at >> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating >> >>> [INFO] [Management$] Inspecting Apache Spark... >> >>> [INFO] [Management$] Apache Spark is installed at >> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5. >> 1-bin-hadoop2.6 >> >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum >> >>> requirement of 1.3.0) >> >>> [INFO] [Management$] Inspecting storage backend connections... >> >>> [INFO] [Storage$] Verifying Meta Data Backend (Source: >> ELASTICSEARCH)... >> >>> [ERROR] [Management$] Unable to connect to all storage backends >> >>> successfully. >> >>> The following shows the error message from the storage backend. >> >>> >> >>> Connection closed >> >>> (org.apache.predictionio.shaded.org.apache.http. >> ConnectionClosedException) >> >>> >> >>> Dumping configuration of initialized storage backend sources. >> >>> Please make sure they are correct. >> >>> >> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME >> -> >> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/ >> elasticsearch-1.4.4, >> >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE >> -> >> >>> elasticsearch >> >>> >> >>> >> >>> Can anyone give me an idea of what I need to fix this issue? Here is >> >>> >> >>> >> >>> # PredictionIO Main Configuration >> >>> # >> >>> # This section controls core behavior of PredictionIO. It is very >> likely >> >>> that >> >>> # you need to change these to fit your site. >> >>> >> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be >> configured. >> >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7 >> >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6 >> >>> >> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar >> >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar >> >>> >> >>> # ES_CONF_DIR: You must configure this if you have advanced >> configuration >> >>> for >> >>> # your Elasticsearch setup. >> >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf >> >>> >> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run >> >>> PredictionIO >> >>> # with Hadoop 2. >> >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf >> >>> >> >>> # HBASE_CONF_DIR: You must configure this if you intend to run >> >>> PredictionIO >> >>> # with HBase on a remote cluster. >> >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf >> >>> >> >>> # Filesystem paths where PredictionIO uses as block storage. >> >>> PIO_FS_BASEDIR=$HOME/.pio_store >> >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines >> >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp >> >>> >> >>> # PredictionIO Storage Configuration >> >>> # >> >>> # This section controls programs that make use of PredictionIO's >> built-in >> >>> # storage facilities. Default values are shown below. >> >>> # >> >>> # For more information on storage configuration please refer to >> >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/ >> >>> >> >>> # Storage Repositories >> >>> >> >>> # Default is to use PostgreSQL >> >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta >> >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH >> >>> >> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event >> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE >> >>> >> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model >> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS >> >>> >> >>> # Storage Data Sources >> >>> >> >>> # PostgreSQL Default Settings >> >>> # Please change "pio" to your database name in >> >>> PIO_STORAGE_SOURCES_PGSQL_URL >> >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and >> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly >> >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc >> >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio >> >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio >> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio >> >>> >> >>> # MySQL Example >> >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc >> >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio >> >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio >> >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio >> >>> >> >>> # Elasticsearch Example >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200 >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http >> >>> # >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/ >> vendors/elasticsearch-5.5.2 >> >>> # Optional basic HTTP auth >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name >> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret >> >>> # Elasticsearch 1.x Example >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >> >>> >> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/ >> vendors/elasticsearch-1.4.4 >> >>> >> >>> # Local File System Example >> >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >> >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models >> >>> >> >>> # HBase Example >> >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase >> >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0 >> >>> >> >>> # AWS S3 Example >> >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3 >> >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket >> >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model >> >>> >> >>> >> >>> Thanks, >> >>> >> >>> Jim >> >>> >> >>> >> >>> -- >> >>> Jim Miller >> >> >> >> >> >> >> > >> >>
