I got it to work by replacing 'es.resource.write'='http://eshost:9200' 'es.resource.node'='http://eshost:9200'
as one of the tbl properties in hive. Thanks for your help Costin. On Friday, March 14, 2014 1:12:00 PM UTC-5, Costin Leau wrote: > > You should not change the configuration in Elasticsearch. By default, ES > binds on all available interfaces - specifying > an Ip:port is likely to restrict access rather than extend it. > > To find out the configuration options in es-hadoop, look no further than > the docs [1]. If the hive job is running on the > same machine as elasticsearch, the default (localhost:9200) should be > enough. As an alternative you can specify the > public ip of your host: x.y.z.w > > As a last step, enable TRACE logging (through log4j) on > org.elasticsearch.hadoop package and see what comes out > including network connectivity. > > By the way, make sure you don't have any firewall or proxy set on your > system which might be picked up automatically by > the JVM. > > [1] > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html > > > On 3/14/2014 6:50 PM, P lva wrote: > > Costin, > > > > About what you asked, > > 1) Hive server is running on the same machine as elasticsearch, > > 2) I get a response when I do a curl http://elasticsearc&hivehost:9200 > > > > > > I feel I'm missing something simple. This is what I've got until now. > > > > 1) First error (connection refused) is because i left the default > settings as is. > > > > I changed the 'network.host' in elasticsearch.yml to the hostname > > > > 2) Second error (cannot discover elasticsearch version) is when I > changed the 'network.host' in elasticsearch.yml to the > > hostname. > > > > Looked at the source code and figured I'm expected to pass eshost:port > to the hive table at table creation. > > ( > https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48) > > > > . > > So I included 'es.resource.write'='http://elasticsearchhost:9200' as > one of the table properties during table creation step. > > > > STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' > > TBLPROPERTIES('es.resource' = 'esdemo/hive', > > 'es.mapping.name'='time:@timestamp', > > 'es.resource.write'='http://eshost:9200'); > > > > Now I get connection refused again. Is that the right way to pass that > information ? > > > > Thanks for you patience and help > > > > > > > > > > On Friday, March 14, 2014 10:33:14 AM UTC-5, Costin Leau wrote: > > > > Without any extra information I'm afraid I can only guess what might > be the issue. > > Make sure you have the latest Elasticsearch 0.90 or 1.x available on > port 9200 with the HTTP/REST port open. > > Also make sure that Hive actually runs on the same machine - not > just the client but also the server (meaning Hadoop > > itself). > > > > You indicate that if you change the network configuration you get an > error regarding the version - this suggests that: > > > > 1. Hive is actually running on a different machine than ES - hence > the network error > > 2. After pointing Hive to the actual ES machine, you get an error > since you're using an old Elasticsearch version > > (0.20) > > > > Cheers, > > > > On 3/14/2014 12:19 AM, P lva wrote: > > > I have a simple query > > > insert into table eslogs select * from eslogs_ext; > > > Hive and elasticsearch are running on the same host. > > > > > > To execute the script I'm following the directions from the link. > > > > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html > > < > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html> > > > > > > > There are two elasticsearch nodes, and they can recognize each > other (as indicated by start up process) , but why would > > > hive not be able to pick them up ? Can you explain what could have > gone wrong ? > > > > > > > > > On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau wrote: > > > > > > What does your Hive script look like? Can you confirm the > ip/address of your Hive and Elasticsearch ? How are you > > > executing the script? > > > The error indicates an error in your network configuration. > > > > > > Cheers, > > > > > > P.S. Feel free to post a gist or whatever it's convenient. > > > > > > On 3/13/2014 10:38 PM, P lva wrote: > > > > Hi, I have few weblogs in a hive table that I'd like to > visualize in kibana. > > > > ES is on the same node as hive server. > > > > > > > > Followed directions from this pagehttp:// > www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html > > < > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html> > > > > < > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html > > < > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>> > > > > > > > > > > I can create a table using esstorage handler, but when I > tried to ingest data into this table I got > > > > > > > > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing > > > > row {***first row of my table**} > > > > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175) > > > > at > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > > > > at > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) > > > > at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > > > at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) > > > > at > java.security.AccessController.doPrivileged(Native Method) > > > > at > javax.security.auth.Subject.doAs(Subject.java:415) > > > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > > > > > > at > org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) > > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > Hive Runtime Error while processing row {*** first row of > > > > my table**} > > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.IOException: Out of nodes and retries; caught exception > > > > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652) > > > > > > at > org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) > > > > at > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842) > > > > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88) > > > > > > at > org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) > > > > at > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842) > > > > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91) > > > > > > at > org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) > > > > at > org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842) > > > > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534) > > > > ... 9 more > > > > Caused by: java.io.IOException: Out of nodes and retries; > caught exception > > > > at > org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81) > > > > at > org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221) > > > > at > org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205) > > > > at > org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209) > > > > at > org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103) > > > > at > org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85) > > > > at > org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60) > > > > > > at > org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165) > > > > > > at > org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50) > > > > > > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638) > > > > > > ... 18 more > > > > Caused by: java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native > Method) > > > > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > > > > > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > > > > > > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > > > > at > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > at java.net.Socket.connect(Socket.java:579) > > > > at java.net.Socket.connect(Socket.java:528) > > > > at java.net.Socket.<init>(Socket.java:425) > > > > at java.net.Socket.<init>(Socket.java:280) > > > > at > > > > > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) > > > > > > at > > > > > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) > > > > > > at > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > > > > at > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > > > > > > at > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > > > > > > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > > > > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > > > > at > org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:160) > > > > > > at > org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:74) > > > > ... 27 more > > > > > > > > Now, I changed the config network.host to the ipadress of > the server. Now when I run hive insert data i get > > > > FAILED: IllegalStateException Cannot discover Elasticsearch > version > > > > > > > > > > > > -- > > > > You received this message because you are subscribed to the > Google Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails > from it, send an email to > > > >[email protected] <javascript:> <mailto: > [email protected] <javascript:> <javascript:> > > <javascript:>>. > > > > To view this discussion on the web visit > > > > > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com> > > > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>> > > > > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer> > > > > > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer > > > < > https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>>. > > > > > > > > > > > For more options, visithttps://groups.google.com/d/optout < > http://groups.google.com/d/optout> <https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>>. > > > > > > -- > > > Costin > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, > send an email to > > >[email protected] <javascript:> <mailto: > [email protected] <javascript:> <javascript:>>. > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com> > > > > > < > https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer > > > < > https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>>. > > > > > > > For more options, visithttps://groups.google.com/d/optout < > https://groups.google.com/d/optout>. > > > > -- > > Costin > > > > -- > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to > > [email protected] <javascript:> <mailto: > [email protected] <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > -- > Costin > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9041befe-dad6-447f-b666-c3384aa32259%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
