There is no such property 'es.resource.node' used in Es-Hadoop.
And es.resource.write is something set internally by the Hive support not to mention it should point to a index/type NOT
host:port.
You most likely want:
es.nodes=eshost:9200
and
es.resource=[index]/[type]
See the previous link that I've listed on the properties supported:
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html
On 3/15/2014 1:48 AM, P lva wrote:
I got it to work by replacing
'es.resource.write'='http://eshost:9200'
'es.resource.node'='http://eshost:9200'
as one of the tbl properties in hive.
Thanks for your help Costin.
On Friday, March 14, 2014 1:12:00 PM UTC-5, Costin Leau wrote:
You should not change the configuration in Elasticsearch. By default, ES
binds on all available interfaces - specifying
an Ip:port is likely to restrict access rather than extend it.
To find out the configuration options in es-hadoop, look no further than
the docs [1]. If the hive job is running on
the
same machine as elasticsearch, the default (localhost:9200) should be
enough. As an alternative you can specify the
public ip of your host: x.y.z.w
As a last step, enable TRACE logging (through log4j) on
org.elasticsearch.hadoop package and see what comes out
including network connectivity.
By the way, make sure you don't have any firewall or proxy set on your
system which might be picked up automatically by
the JVM.
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html>
On 3/14/2014 6:50 PM, P lva wrote:
> Costin,
>
> About what you asked,
> 1) Hive server is running on the same machine as elasticsearch,
> 2) I get a response when I do a curlhttp://elasticsearc&hivehost:9200
>
>
> I feel I'm missing something simple. This is what I've got until now.
>
> 1) First error (connection refused) is because i left the default
settings as is.
>
> I changed the 'network.host' in elasticsearch.yml to the hostname
>
> 2) Second error (cannot discover elasticsearch version) is when I changed
the 'network.host' in elasticsearch.yml to the
> hostname.
>
> Looked at the source code and figured I'm expected to pass eshost:port to
the hive table at table creation.
>
(https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48
<https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48>)
> .
> So I included 'es.resource.write'='http://elasticsearchhost:9200
<http://elasticsearchhost:9200>' as one of the table properties during table
creation
step.
>
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = 'esdemo/hive',
> 'es.mapping.name
<http://es.mapping.name>'='time:@timestamp',
> 'es.resource.write'='http://eshost:9200');
>
> Now I get connection refused again. Is that the right way to pass that
information ?
>
> Thanks for you patience and help
>
>
>
>
> On Friday, March 14, 2014 10:33:14 AM UTC-5, Costin Leau wrote:
>
> Without any extra information I'm afraid I can only guess what might
be the issue.
> Make sure you have the latest Elasticsearch 0.90 or 1.x available on
port 9200 with the HTTP/REST port open.
> Also make sure that Hive actually runs on the same machine - not just
the client but also the server (meaning Hadoop
> itself).
>
> You indicate that if you change the network configuration you get an
error regarding the version - this suggests that:
>
> 1. Hive is actually running on a different machine than ES - hence
the network error
> 2. After pointing Hive to the actual ES machine, you get an error
since you're using an old Elasticsearch version
> (0.20)
>
> Cheers,
>
> On 3/14/2014 12:19 AM, P lva wrote:
> > I have a simple query
> > insert into table eslogs select * from eslogs_ext;
> > Hive and elasticsearch are running on the same host.
> >
> > To execute the script I'm following the directions from the link.
>
>http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>>
> >
> > There are two elasticsearch nodes, and they can recognize each
other (as indicated by start up process) , but why would
> > hive not be able to pick them up ? Can you explain what could have
gone wrong ?
> >
> >
> > On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau wrote:
> >
> > What does your Hive script look like? Can you confirm the
ip/address of your Hive and Elasticsearch ? How are you
> > executing the script?
> > The error indicates an error in your network configuration.
> >
> > Cheers,
> >
> > P.S. Feel free to post a gist or whatever it's convenient.
> >
> > On 3/13/2014 10:38 PM, P lva wrote:
> > > Hi, I have few weblogs in a hive table that I'd like to
visualize in kibana.
> > > ES is on the same node as hive server.
> > >
> > > Followed directions from this
pagehttp://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>>
> >
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html>>>
> > >
> > > I can create a table using esstorage handler, but when I
tried to ingest data into this table I got
> > >
> > > Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
> > > row {***first row of my table**}
> > > at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
> > > at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > > at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> > > at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> > > at
java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> > > at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row {*** first row of
> > > my table**}
> > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: Out of nodes and retries; caught exception
> > > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
> > > ... 9 more
> > > Caused by: java.io.IOException: Out of nodes and retries;
caught exception
> > > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
> > > at
org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
> > > at
org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85)
> > > at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60)
> > > at
org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165)
> > > at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50)
> > > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
> > > ... 18 more
> > > Caused by: java.net.ConnectException: Connection refused
> > > at java.net.PlainSocketImpl.socketConnect(Native
Method)
> > > at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> > > at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> > > at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > > at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > at java.net.Socket.connect(Socket.java:579)
> > > at java.net.Socket.connect(Socket.java:528)
> > > at java.net.Socket.<init>(Socket.java:425)
> > > at java.net.Socket.<init>(Socket.java:280)
> > > at
> > >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
> > > at
> > >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
> > > at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> > > at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> > > at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> > > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> > > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> > > at
org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:160)
> > > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:74)
> > > ... 27 more
> > >
> > > Now, I changed the config network.host to the ipadress of the
server. Now when I run hive insert data i get
> > > FAILED: IllegalStateException Cannot discover Elasticsearch
version
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from
it, send an email to
> > >[email protected] <javascript:>
<mailto:[email protected] <javascript:>
<javascript:>
> <javascript:>>.
> > > To view this discussion on the web visit
> >
>https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>>
> >
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>>>
> > >
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>
>
> >
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
>
> >
> > > For more options, visithttps://groups.google.com/d/optout
<http://groups.google.com/d/optout> <http://groups.google.com/d/optout
<http://groups.google.com/d/optout>> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>>.
> >
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it,
send an email to
> >[email protected] <javascript:>
<mailto:[email protected] <javascript:>
<javascript:>>.
> > To view this discussion on the web visit
>
>https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com>>
> >
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> > For more options, visithttps://groups.google.com/d/optout
<http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
>
> --
> Costin
>
> --
> You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
>[email protected] <javascript:>
<mailto:[email protected] <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9041befe-dad6-447f-b666-c3384aa32259%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/9041befe-dad6-447f-b666-c3384aa32259%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53240E6B.5020709%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
