You should not change the configuration in Elasticsearch. By default, ES binds on all available interfaces - specifying
an Ip:port is likely to restrict access rather than extend it.
To find out the configuration options in es-hadoop, look no further than the docs [1]. If the hive job is running on the
same machine as elasticsearch, the default (localhost:9200) should be enough. As an alternative you can specify the
public ip of your host: x.y.z.w
As a last step, enable TRACE logging (through log4j) on org.elasticsearch.hadoop package and see what comes out
including network connectivity.
By the way, make sure you don't have any firewall or proxy set on your system which might be picked up automatically by
the JVM.
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html
On 3/14/2014 6:50 PM, P lva wrote:
Costin,
About what you asked,
1) Hive server is running on the same machine as elasticsearch,
2) I get a response when I do a curl http://elasticsearc&hivehost:9200
I feel I'm missing something simple. This is what I've got until now.
1) First error (connection refused) is because i left the default settings as
is.
I changed the 'network.host' in elasticsearch.yml to the hostname
2) Second error (cannot discover elasticsearch version) is when I changed the
'network.host' in elasticsearch.yml to the
hostname.
Looked at the source code and figured I'm expected to pass eshost:port to the
hive table at table creation.
(https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48)
.
So I included 'es.resource.write'='http://elasticsearchhost:9200' as one of the
table properties during table creation step.
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'esdemo/hive',
'es.mapping.name'='time:@timestamp',
'es.resource.write'='http://eshost:9200');
Now I get connection refused again. Is that the right way to pass that
information ?
Thanks for you patience and help
On Friday, March 14, 2014 10:33:14 AM UTC-5, Costin Leau wrote:
Without any extra information I'm afraid I can only guess what might be the
issue.
Make sure you have the latest Elasticsearch 0.90 or 1.x available on port
9200 with the HTTP/REST port open.
Also make sure that Hive actually runs on the same machine - not just the
client but also the server (meaning Hadoop
itself).
You indicate that if you change the network configuration you get an error
regarding the version - this suggests that:
1. Hive is actually running on a different machine than ES - hence the
network error
2. After pointing Hive to the actual ES machine, you get an error since
you're using an old Elasticsearch version
(0.20)
Cheers,
On 3/14/2014 12:19 AM, P lva wrote:
> I have a simple query
> insert into table eslogs select * from eslogs_ext;
> Hive and elasticsearch are running on the same host.
>
> To execute the script I'm following the directions from the link.
>http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>
>
> There are two elasticsearch nodes, and they can recognize each other (as
indicated by start up process) , but why would
> hive not be able to pick them up ? Can you explain what could have gone
wrong ?
>
>
> On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau wrote:
>
> What does your Hive script look like? Can you confirm the ip/address
of your Hive and Elasticsearch ? How are you
> executing the script?
> The error indicates an error in your network configuration.
>
> Cheers,
>
> P.S. Feel free to post a gist or whatever it's convenient.
>
> On 3/13/2014 10:38 PM, P lva wrote:
> > Hi, I have few weblogs in a hive table that I'd like to visualize
in kibana.
> > ES is on the same node as hive server.
> >
> > Followed directions from this
pagehttp://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>>
> >
> > I can create a table using esstorage handler, but when I tried to
ingest data into this table I got
> >
> > Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
> > row {***first row of my table**}
> > at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
> > at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> > at java.security.AccessController.doPrivileged(Native
Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> > at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row {*** first row of
> > my table**}
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: Out of nodes and retries; caught exception
> > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652)
> > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
> > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
> > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
> > ... 9 more
> > Caused by: java.io.IOException: Out of nodes and retries; caught
exception
> > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
> > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
> > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
> > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
> > at
org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
> > at
org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85)
> > at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60)
> > at
org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165)
> > at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50)
> > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
> > ... 18 more
> > Caused by: java.net.ConnectException: Connection refused
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> > at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> > at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > at java.net.Socket.connect(Socket.java:579)
> > at java.net.Socket.connect(Socket.java:528)
> > at java.net.Socket.<init>(Socket.java:425)
> > at java.net.Socket.<init>(Socket.java:280)
> > at
> >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
> > at
> >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
> > at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> > at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> > at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> > at
org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:160)
> > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:74)
> > ... 27 more
> >
> > Now, I changed the config network.host to the ipadress of the
server. Now when I run hive insert data i get
> > FAILED: IllegalStateException Cannot discover Elasticsearch version
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it,
send an email to
> >[email protected] <javascript:>
<mailto:[email protected] <javascript:>
<javascript:>>.
> > To view this discussion on the web visit
>
>https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>>
> >
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> > For more options, visithttps://groups.google.com/d/optout
<http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
>
> --
> Costin
>
> --
> You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
>[email protected] <javascript:>
<mailto:[email protected] <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53234670.5040201%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
