Yes, I use /etc/hosts Unless for HBase's Master and RegionServer which require a DNS for resolving hostname.
I encountered an issue, which is that Zookeeper doesn't recognize the hdfs:// protocol too. It instead create a directory named hdfs:\ 2017-04-10 19:32 GMT+04:00 Pat Ferrel <[email protected]>: > RE Elasticsearch, there is no need to have the clustername and hostname > the same, in fact I have never done this and we run larger clusters of ES. > Are you using /etc/hosts or something other than DNS to resolve the > hostname or some other mechanism? > > I guess if it’s working, it’s not an immediate issue but just wanted to > make that clear. > > > On Apr 10, 2017, at 8:27 AM, Marius Rabenarivo <[email protected]> > wrote: > > Yes, I set the clustername in Elasticsearch config too. > > Thank you for your advices about fields. > > 2017-04-10 19:07 GMT+04:00 Pat Ferrel <[email protected]>: > >> Do you have the clustername set in Elasticsearch config also? It is >> changed in /usr/local/elasticsearch/config/elasticsearch.yml Setting >> this in both places will decouple server name from cluster name. >> Unfortunately changing a cluster name will call for a complete reset of >> PredictionIO data since it will loose track of engine metadata. I have >> found it easiest to export/backup then delete all apps and re-install or >> erase any table in hbase and elasticsearch. You have to get the clustername >> right before you start because it’s a pain to change. >> >> Also the fields must correspond to arrays of strings, not an array of >> true or false, an array of one element, a string of “T” or “F” would work >> for this. For fields the property must be attached to all items by using a >> $set event, then your visible and sponsored will work more like what you >> expect. They are not working at all now. >> >> >> On Apr 9, 2017, at 10:43 PM, Marius Rabenarivo < >> [email protected]> wrote: >> >> Hello, >> >> Find attached the pio-env.sh and engine.json >> >> that we are using. >> >> 2017-04-08 18:46 GMT+04:00 Marius Rabenarivo <[email protected]> >> : >> >>> I'll share them monday. >>> >>> 2017-04-07 22:23 GMT+04:00 Pat Ferrel <[email protected]>: >>> >>>> We use a clustername different from hostname all the time. In fact if >>>> you have more than one node you will have to make them different since >>>> there are multiple hostnames and one clustername. >>>> >>>> The clustername should be something unique to your application because >>>> ES has the sometimes helpful, but often maddeningly bad and hidden, habit >>>> of merging nodes into a cluster by clustername. For instance the default >>>> for ES is to merge all nodes on a LAN into one cluster. >>>> >>>> Anyway can you share your engine.json and the result of curl with >>>> exactly the same hostname? You need to add the clustername to pio-env.sh >>>> too so can you share that as well? >>>> >>>> >>>> On Apr 7, 2017, at 12:25 AM, Marius Rabenarivo < >>>> [email protected]> wrote: >>>> >>>> It works when using the default port 9200 and by changing the >>>> Elasticsearch cluster name to the same as the hostname as suggested here >>>> >>>> http://stackoverflow.com/questions/25885411/elasticsearch-er >>>> ror-failed-to-connect-to-master-no-route-to-host >>>> >>>> 2017-04-07 9:57 GMT+04:00 Marius Rabenarivo <[email protected] >>>> >: >>>> >>>>> I tried to run the command >>>>> >>>>> curl -X GET http://engine-server:9200 >>>>> >>>>> and I'm getting a JSON response but when I'm running the command >>>>> >>>>> pio train >>>>> >>>>> I got >>>>> >>>>> org.elasticsearch.hadoop.rest.EsHadoopTransportException: >>>>> java.net.NoRouteToHostException: >>>>> No route to host (Host unreachable) >>>>> >>>>> exception >>>>> >>>>> 2017-04-07 9:18 GMT+04:00 Marius Rabenarivo <mariusrabenarivo@g >>>>> mail.com>: >>>>> >>>>>> I tried using these configurations in the sparkConf section of >>>>>> engine.json >>>>>> It has the same effect as the code I mentioned below but I still >>>>>> getting the same exception. >>>>>> >>>>>> 2017-04-07 5:03 GMT+04:00 Pat Ferrel <[email protected]>: >>>>>> >>>>>>> BTW the docs for sparkConf are here: http://actionml.com/docs >>>>>>> /ur_config#ur-spark-conf >>>>>>> >>>>>>> >>>>>>> On Apr 6, 2017, at 4:52 PM, Pat Ferrel <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> PIO in the past used the “transport client” of Elasticsearch, which >>>>>>> is on port 9300 by default, The UR uses the REST client for some things >>>>>>> like rapid writes to ES using Spark using a lib supplied by >>>>>>> Elasticsearch. >>>>>>> >>>>>>> So, Spark needs to have config that the Elasticsearch library can >>>>>>> read. This can be done most easily by putting it into a sparkConf >>>>>>> section >>>>>>> of engine.json >>>>>>> >>>>>>> Like this: >>>>>>> >>>>>>> "sparkConf": { >>>>>>> "spark.serializer": "org.apache.spark.serializer.K >>>>>>> ryoSerializer", >>>>>>> "spark.kryo.registrator": "org.apache.mahout.sparkbindings.io >>>>>>> .MahoutKryoRegistrator", >>>>>>> "spark.kryo.referenceTracking": "false", >>>>>>> "spark.kryoserializer.buffer": "300m", >>>>>>> "es.index.auto.create": “true”, >>>>>>> “es.nodes”: “node1,node2” <========== add this >>>>>>> }, >>>>>>> >>>>>>> The list is comma separated host ip addresses of DNS names so no >>>>>>> spaces. If you are using the default port 9200 you don’t need to specify >>>>>>> it. These values are defined by Elasticsearch so see their docs if you >>>>>>> have >>>>>>> any question about what can be configured in this way. For instance you >>>>>>> can >>>>>>> also specify password and username in this manner. >>>>>>> >>>>>>> >>>>>>> On Apr 6, 2017, at 4:27 PM, Marius Rabenarivo < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I try to run training from a remote machine. >>>>>>> >>>>>>> It fails in later stage due to this exception : >>>>>>> >>>>>>> org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: >>>>>>> Connection error (check network and/or proxy settings)- all nodes >>>>>>> failed; >>>>>>> tried [[engine-server:9300]] >>>>>>> >>>>>>> According to the stack trace the error is coming from this line : >>>>>>> >>>>>>> https://github.com/actionml/universal-recommender/blob/maste >>>>>>> r/src/main/scala/EsClient.scala#L179 >>>>>>> >>>>>>> It was using a default configuration to access ES. >>>>>>> >>>>>>> I tried to replace this line with : >>>>>>> >>>>>>> >>>>>>> val elasticSearchConf = Storage.getConfig("ELASTICSEARCH").get >>>>>>> >>>>>>> val conf = Map( >>>>>>> "es.mapping.id" -> "id", >>>>>>> "es.nodes" -> elasticSearchConf.properties.get("HOSTS").head, >>>>>>> "es.port" -> elasticSearchConf.properties.get("PORTS").head >>>>>>> ) >>>>>>> >>>>>>> indexRDD.saveToEs(newIndexURI, conf) >>>>>>> >>>>>>> but I still getting the same error. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Marius >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "actionml-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To view this discussion on the web visit https://groups.google.co >>>>>>> m/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6Na >>>>>>> V10Eh1S4_t1Ub4PcA%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "actionml-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To view this discussion on the web visit https://groups.google.co >>>>>>> m/d/msgid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667 >>>>>>> %40occamsmachete.com >>>>>>> <https://groups.google.com/d/msgid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667%40occamsmachete.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "actionml-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.co >>>> m/d/msgid/actionml-user/CAC-ATVGHNYHJ-V78B1knbpqWGn_cpkvYCC_ >>>> %2BW3Tmdw6kywcSTw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/actionml-user/CAC-ATVGHNYHJ-V78B1knbpqWGn_cpkvYCC_%2BW3Tmdw6kywcSTw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "actionml-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/actionml-user/CAC-ATVG8madmx19TqijGra4n6TxDSnxvziB4RNggn >> vAosB7Etg%40mail.gmail.com >> <https://groups.google.com/d/msgid/actionml-user/CAC-ATVG8madmx19TqijGra4n6TxDSnxvziB4RNggnvAosB7Etg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> <engine.json><pio-env.sh> >> >> > > -- > You received this message because you are subscribed to the Google Groups > "actionml-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/actionml-user/CAC-ATVESU16R_w_Hs%2B49a7unJmpUdjghASDjXRiYhdZRrE > AggQ%40mail.gmail.com > <https://groups.google.com/d/msgid/actionml-user/CAC-ATVESU16R_w_Hs%2B49a7unJmpUdjghASDjXRiYhdZRrEAggQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > >
