I've tried this :
create external table es_word(word string,cnt bigint)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'secure/secure',
'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200>',
'es.net.http.auth.user'='myysername',
'es.net.http.auth.pass'='mypassword');
INSERT OVERWRITE TABLE es_word SELECT s.word, s.cnt FROM word_countstest s;
But the job is terminated with an authentication error :
Any idea why? I've entered the thr right esuser and pwd that I've defined and
am using to access to my ES cluster !
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
{"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:
AuthenticationException[missing authentication token for REST request [/]]
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:286)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:572)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:
AuthenticationException[missing authentication token for REST request [/]]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:319)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:284)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:288)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:117)
at org.elasticsearch.hadoop.rest.RestClient.esVersion(RestClient.java:364)
at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:86)
at org.elasticsearch.hadoop.hive.HiveUtils.init(HiveUtils.java:145)
at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:93)
at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:42)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
... 20 more
ᐧ
On Thu, Apr 2, 2015 at 9:59 PM, BEN SALEM Omar <[email protected]
<mailto:[email protected]>> wrote:
Hey Cosin,
And when It comes to the marvel?
Can you see why It's not seeing my indices?
Thanks,
On Thu, Apr 2, 2015 at 9:46 PM, Costin Leau <[email protected]
<mailto:[email protected]>> wrote:
The configuration example is correct - one can pass any configuration
property there and it will be picked up
as explained here [1]
Additionally, to verify one can turn on logging [2] to see the
connection is properly authenticated
[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/pig.html#pig-configuration
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html
On 4/2/15 9:21 PM, Jay Modi wrote:
Still not sure what's going on with Marvel, unfortunately. No idea
if this will help, but have you tried
clearing your
browser cache or using a incognito window?
I think something like this:
TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200>
<http://192.168.238.130:9200/>',
'es.net.http.auth.user'='omar',
'es.net.http.auth.pass'='omar')
I copy/pasted the wrong names earlier. I'm not a hadoop user but I
believe that should work based off of
the following
documentation:
http://www.elastic.co/guide/en/shield/current/hadoop.html
http://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#hive-configuration
On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar
wrote:
Jay,
when it comes to adding the .user and .password in the
tblproperties
Do you know what exactly to do?
let's suppose that my admin esuser is omar and pwd omar
what should I do?
TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200>
<http://192.168.238.130:9200>',
and now what to write);
and is there any further config to do?
Thanks for your time
ᐧ
On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar <[email protected]
<mailto:[email protected]>
<javascript:>> wrote:
Yes I had,
I used to manage my cluster with marvel perfectly and all
the indices used to show up.
But now, none of them seem to be found due to time span;
and even when I removed the shield this
"porblem" occured.
I don't see how to "restore" the previous configuration.
Any idea?
ᐧ
On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi <[email protected]
<mailto:[email protected]> <javascript:>>
wrote:
There should not be anything extra necessary to work
with Marvel. The browser prompted you for
credentials
and you entered admin credentials, correct?
On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN
SALEM Omar wrote:
Or not, that wenti as I'd like to go :))
Here is the last remaining issue :
ᐧ
On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar
<[email protected]
<mailto:[email protected]>> wrote:
That would be a problem !
Last question (as for now :p ), how to change
the marvel timepicker? I can't see any
indice of
mine(as I already told you)
And hey, thank you for your cooperation Jay
ᐧ
On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi
<[email protected] <mailto:[email protected]>> wrote:
Hi Omar,
I think you need to add "es.net.http.auth.user" and
"es.net.http.auth_password" as
additional TBLPROPERTIES. You will need to
using elasticsearch Hadoop 2.1 beta 3
for this to
work as far as I know.
-Jay
On Thursday, April 2, 2015 at 11:18:12 AM
UTC-4, BEN SALEM Omar wrote:
Hi Jay,
Now that I have shield in both ES and
Kibana, and I want to load some data
from my hadoop
cluster to ES.
I usually do this :
....
....
create external table
es_word_count(word string, count int)
STORED BY
'org.elasticsearch.hadoop.__hive__.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'test/test',
'es.nodes'='192.168.238.130 <http://192.168.238.130>:__92__00');
INSERT OVERWRITE TABLE es_word_count
SELECT s.word, s.count FROM word_counts1 s;
what should I do or add to access to my
ES cluster as the esuser I've created?
Thanks
ᐧ
On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi
<[email protected]
<mailto:[email protected]>> wrote:
Hi Omar,
Have you configured Kibana and
Marvel to work with Shield? For Kibana 4,
please see
http://www.elastic.co/__guid__e/en/kibana/current/__production__.html#configuring-__kibana-__shield
<http://www.elastic.co/guide/en/kibana/current/production.html#configuring-kibana-shield> and
http://www.elastic.co/__guid__e/en/shield/current/___shield___with_kibana_4.html
<http://www.elastic.co/guide/en/shield/current/_shield_with_kibana_4.html>. For Marvel,
see
http://www.elastic.co/__guid__e/en/shield/current/__marvel.__html
<http://www.elastic.co/guide/en/shield/current/marvel.html>
-Jay
On Thursday, April 2, 2015 at
6:45:55 AM UTC-4, BEN SALEM Omar wrote:
Hi Jay,
I have a question, now that
I've added an esuser as an admin, I can
not run my
Kibana anymore,
Any idea on what to do to
overcome this?
Another basic question, when I
open the marvel, it shows no indice
because the
sample of data I'm working with is in
"2014", how to change my time
span to see my
indices again?
btwn, before I install the
shield, Marvel used to recognize my
timestamp by default
ᐧ
Thank you,
Omar,
On Wed, Apr 1, 2015 at 8:22 PM,
BEN SALEM Omar <[email protected]
<mailto:[email protected]>> wrote:
When I try to add an esuser
and grant him a role:
bin/shield/esusers useradd rdeniro-p taxidriver-r admin
It says that there is an error in the
"esusers" file in the
elef['$ES_INCLUDE"]
line, I think the line 31
or 38, I forgot.
In the line above, there is a
$...done: which ends with ":", if
you change it to
done; with ";" , the error
doesn't show up anymore.
I'm installing the whole
thing in the Hortonworks Sandbox 2.0 vm
On Wed, Apr 1, 2015 at 8:14
PM, Jay Modi <[email protected]
<mailto:[email protected]>> wrote:
I'm glad you were able to fix this issue. I have a few questions
that may
help us ensure this fix is pulled back into Shield. What OS and
Shell are
you using? Which line in the script did you change exactly? I'm
looking at
the script right now and I don't see any instance of "done:"
Just an FYI, there is another way to handle the configuration
directory
issue that is documented in the configuring your environment
section of the
shield guide:
http://www.elastic.co/guide/__en____/shield/current/getting-__starte____d.html#_configuring___your___envir__onment
<http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment>
On Wednesday, April 1,
2015 at 2:07:28 PM UTC-4, BEN SALEM
Omar wrote:
Ok, I've managed to solve my own problem.
First of all things, if you want to just remove the shield plugin
run
under the elasticsearch :
$ bin/plugin --remove shield
Now, to the syntax error in the elef line in the "esusers",
generally,
when there is such an error, it's related to things before that
line.
In fact there was something like this :
$....done:
correction :
$....done;
Now, we could add users within shield and grunt them the desired
role.
But, even if I've done it, I couldn't access to my ES cluster
with the
"new born user"; I think because my ES is running as a service
and not
starting with bin/elasticsearch, that lead to "misunderstanding."
The solution I found was to transfer the
/usr/share/elasticsearch/__confi______g/shield under
/etc/elasticsearch
$ln -s /usr/share/elasticsearch/__confi______g/shield/
/etc/elasticsearch
And that was it !
Hope that could be of a help for anyone else facing the same
problem !
On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar
wrote:
I wanted to add authentication to my ES, thus I downloaded
the shield :
https://www.elastic.co/__downloa______ds/shield
<https://www.elastic.co/downloads/shield>
Then I've tried to add a user to give him the admin role as
it's
mentionned here :
http://www.elastic.co/guide/__en______/shield/current/getting-__starte______d.html
<http://www.elastic.co/guide/en/shield/current/getting-started.html>
(I've added it with root, cause I start my es with root..)
But when I try to do so, an error occure saying that there is
a
syntax error in the 38 line "the elef line"
elef[$ES_INCLUDE] bla bla
I've tried many time :
bin/shield/esusers useradd omar -p omar -r admin
bin/shield/esusers useradd esomar -p omar -r admin
bin/shield/esusers useradd es_omar -p omar -r admin
bin/shield/esusers useradd ES_omar -p omar -r admin
bin/shield/esusers useradd omar -r admin
bin/shield/esusers useradd esomar -r admin
bin/shield/esusers useradd es_omar -r admin
bin/shield/esusers useradd ES_omar -r admin
But the same error kept on jumping !
I said enough is enough and wanted to keep on doing wwhat I
was
doing, tried to run Kibana, and this is what happened !
I've then tried to open my ES marvel plugin, but there is now
an
authentication box
I've tried ALL the combinations above, but none of them
seemed to
work !!
I'm really preoccupied by this note :
To ensure that Elasticsearch can read the user and role
information
at startup, run |esusers useradd| as the same user you use to
run
Elasticsearch. Running the command as root or some other user
will
update the permissions for the |users| and
|users_roles|__file______s and prevent Elasticsearch from
accessing
them.
The thing is that I start my es and kibana with the root..
Please HELP !!
--
You received this message because you are subscribed to a topic in
the
Google Groups "elasticsearch" group.
To unsubscribe from
this topic, visit
https://groups.google.com/d/__to____pic/elasticsearch/sEZpCO7E___B0/____unsubscribe
<https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from
this group and all its topics, send an
email to
elasticsearc...@__googlegroups.__c__om.
To view this discussion
on the web visit
https://groups.google.com/d/__ms____gid/elasticsearch/9f0ebe63-__cc8____3-4c0b-8ee9-1514e51f77f6%__40goo____glegroups.com
<https://groups.google.com/d/msgid/elasticsearch/9f0ebe63-cc83-4c0b-8ee9-1514e51f77f6%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit
https://groups.google.com/d/__op____tout
<https://groups.google.com/d/optout>.
--
You received this message because
you are subscribed to a topic in the
Google Groups
"elasticsearch" group.
To unsubscribe from this topic,
visit
https://groups.google.com/d/__to__pic/elasticsearch/sEZpCO7E___B0/__unsubscribe
<https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from this group and
all its topics, send an email to
elasticsearc...@__googlegroups.__com.
To view this discussion on the web
visit
https://groups.google.com/d/__ms__gid/elasticsearch/35767d7f-__45d__7-4113-a478-499d44abcf38%__40goo__glegroups.com
<https://groups.google.com/d/msgid/elasticsearch/35767d7f-45d7-4113-a478-499d44abcf38%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit
https://groups.google.com/d/__op__tout
<https://groups.google.com/d/optout>.
--
You received this message because you are
subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/sEZpCO7E___B0/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from this group and all its
topics, send an email to
elasticsearc...@__googlegroups.com
<http://googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/__msgid/elasticsearch/579d2139-__a75f-4c17-a03e-51e0f4830faf%__40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/579d2139-a75f-4c17-a03e-51e0f4830faf%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit
https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to
a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from this group and all its topics, send
an email to
[email protected]
<mailto:[email protected]>
<javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9622bb08-27c8-45b0-a892-b7412ca8fdff%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/9622bb08-27c8-45b0-a892-b7412ca8fdff%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
[email protected]
<mailto:elasticsearch%[email protected]>
<mailto:[email protected]
<mailto:elasticsearch%[email protected]>>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e830a27c-a73a-447f-bbbf-58c7a9e31957%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/e830a27c-a73a-447f-bbbf-58c7a9e31957%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
[email protected]
<mailto:elasticsearch%[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/551D9CA9.6040809%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[email protected] <mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGgBrpxW801AgUXqs_WAZEqgYe3KjZiZ2sOy6dja-J4nXQ5Cuw%40mail.gmail.com
<https://groups.google.com/d/msgid/elasticsearch/CAGgBrpxW801AgUXqs_WAZEqgYe3KjZiZ2sOy6dja-J4nXQ5Cuw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
