Hi, Thanks for the help and you are right, REST API was not working due to the problem I found below.
The problem and solution, just in case someone else encounters the same: The problem was we are using a hosted ES as a service and while writing the data to index, it first discovers all the ES nodes and then try to write. The process of discovery gives it different hosts and ports then the one provided by me in config. Hence in in UR template (build.sbt), I needed to update: "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.2" to "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.2.0" Also in sparkConf in engine.json, I needed to provide: "es.nodes.wan.only": true Ofcourse, The alternate solution would be to get the ES Cluster and UR running in same network but it is not possible for us as ES is outsourced and UR is in aws. Relevant Page: https://www.elastic.co/guide/en/elasticsearch/hadoop/ 2.2/cloud.html Regards Harsh Mathur harshmathur.1...@gmail.com *“Perseverance is the hard work you do after you get tired of doing the hard work you already did."* On Fri, Nov 25, 2016 at 5:05 AM, Pat Ferrel <p...@occamsmachete.com> wrote: > I think the index is created using the transport client so requires no > auth and happens before the writing starts, which uses the REST API. So the > REST API seems to not be working at all. > > Check that the Spark Executor can connect to REST with your config. Also > include "es.index.auto.create": "true" > > > > On Nov 23, 2016, at 6:56 AM, Harsh Mathur <harshmathur.1...@gmail.com> > wrote: > > Hi, > The problem is in training phase of the UR template, the model > (correlators), is built, but unable to persist in elastic search. The > connectivity is there because indices are getting created, though no data > is being pushed. Help appreciated. > > My SparkConf is as follows: > "es.nodes": "hostname", > "es.port": port, > "es.net.http.auth.user": "username", > "es.net.http.auth.pass": "password", > "es.net.ssl": true > > > However, I can see a new index getting created but no data being pushed > (timeout exception) > > https://basic_auth@host:port/_cat/indices?v > green open urindex_1479911005241 4 0 0 0 > 460b 460b > green open pio_meta 4 0 19 1 > 210.8kb 210.8kb > green open urindex_1479909625467 4 0 0 0 > 460b 460b > green open urindex_1479908293213 4 0 0 0 > 460b 460b > green open urindex_1479911811014 4 0 0 0 > 460b 460b > green open urindex_1479910315092 4 0 0 0 > 460b 460b > > > As you can see after every retry, I can see a new index. > > The Exception I get is: > > [INFO] [URAlgorithm] Correlators created now putting into URModel > [INFO] [URModel] Converting cooccurrence matrices into correlators > [INFO] [URModel] Group all properties RDD > [INFO] [URModel] ES fields[10]: List(available, id, defaultRank, > categories, expires, popRank, purchase, countries, view, date) > [Stage 103:> (0 + > 2) / 5][INFO] [HttpMethodDirector] I/O exception > (java.net.ConnectException) caught when processing request: Connection > timed out (Connection timed out) > [INFO] [HttpMethodDirector] Retrying request > [INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException) > caught when processing request: Connection timed out (Connection timed out) > [INFO] [HttpMethodDirector] Retrying request > > > > > Regards > Harsh Mathur > harshmathur.1...@gmail.com > > *“Perseverance is the hard work you do after you get tired of doing the > hard work you already did."* > >
