Ok, let me repeat: running a `kinit` in your local shell has *no
bearing* on what Accumulo is doing. This is fundamentally not how it
works. There are libraries in the JDK which perform the login with the
KDC using the keytab you provide in accumulo-site.xml. Accumulo is not
using the ticket cache which your `kinit` creates.
</rant>
You should see a message in the log stating that the Kerberos login
happened (or didn't). The server should exit if it fails to log in (but
I don't know if I've actively tested that). Do you see this message?
Does it say you successfully logged in (and the principal you logged in as)?
roman.drap...@baesystems.com wrote:
Ok, there is some progress. So these issues were definitely related to VFS
classloader - now works both on the client and master - so I guess a bug is
found.
And it looks like there is a very similar issue related to instance_id
On the slaves (does not matter whether I do kinit hdfs or not) I always receive
when I start the node:
2016-01-26 20:36:41,744 [tserver.TabletServer] ERROR: Uncaught
exception in TabletServer.main, exiting
java.lang.RuntimeException: Can't tell if Accumulo is initialized;
can't read instance id at
hdfs://cr-platform-qa23-01.cyberreveal.local:8020/accumulo/instance_id
On the master I can see the same issue when I do bin/stop-all.sh without kinit
hdfs and it disappears if I have a hdfs ticket.
I tried both: hadoop fs -chown -R accumulo:hdfs /accumulo and hadoop fs -chown
-R accumulo:accumulo /accumulo - same behavior
Any thoughts please?
-----Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: 26 January 2016 20:08
To: user@accumulo.apache.org
Subject: Re: Accumulo and Kerberos
The normal classloader (on the local filesystem) which is configured out of the
box.
roman.drap...@baesystems.com wrote:
Hi Josh,
I can confirm that issue on the master is related to VFS classloader!
Commented out classloader and now it works without kinit. So it seems it tries
loading classes before Kerberos authentication happened. What classloader
should I use instead?
Regards,
Roman
-----Original Message-----
From: roman.drap...@baesystems.com
[mailto:roman.drap...@baesystems.com]
Sent: 26 January 2016 19:43
To: user@accumulo.apache.org
Subject: RE: Accumulo and Kerberos
Hi Josh,
Two quick questions.
1) What should I use instead of HDFS classloader? All examples seem to be from
hdfs.
2) Whan 1.7.1 release is scheduled for (approx.) ?
Regards,
Roman
-----Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: 26 January 2016 19:01
To: user@accumulo.apache.org
Subject: Re: Accumulo and Kerberos
I would strongly recommend that you do not use the HDFS classloader. It is
known to be very broken in what you download as 1.7.0. There are a number of
JIRA issues about this which stem from a lack of a released commons-vfs2-2.1.
That being said, I have not done anything with running Accumulo out of HDFS
with Kerberos enabled. AFAIK, you're in untraveled waters.
re: the renewal bug: When the ticket expires, the Accumulo service
will die. Your options are to deploy a watchdog process that would
restart the service, download the fix from the JIRA case and rebuild
Accumulo yourself, or build 1.7.1-SNAPSHOT from our codebase. I would
recommend using 1.7.1-SNAPSHOT as it should be the least painful
(1.7.1-SNAPSHOT now is likely to not change significantly from what is
ultimately released as 1.7.1)
roman.drap...@baesystems.com wrote:
Hi Josh,
Yes, will do. Just in the meantime - I can see a different issue on slave
nodes. If I try to start in isolation (bin/start-here.sh) with or without doing
kinit I always see the error below.
2016-01-26 18:31:13,873 [start.Main] ERROR: Problem initializing the
class loader java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main.getClassLoader(Main.java:68)
at org.apache.accumulo.start.Main.main(Main.java:52)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not determine the type of file
"hdfs://<hostname>/platform/lib/.*.jar".
at
org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:1522)
at
org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:489)
at
org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.resolve(AccumuloVFSClassLoader.java:143)
at
org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.resolve(AccumuloVFSClassLoader.java:121)
at
org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.getClassLoader(AccumuloVFSClassLoader.java:211)
... 6 more
Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE
authentication is not enabled. Available:[TOKEN, KERBEROS]
I guess it might be different to what I observe on the master node. If I don't
get ticket explicitly, I get the error mentioned in the previous email. However
if do (and it does not matter for what user I have a ticket now - whether it's
accumulo, hdfs or hive) - it works. So I started to think, maybe the problem
related to some action (for example to vfs as per above) that tries to access
HDFS before doing a proper authentication with Kerberos? Any ideas?
Also, if we go live with 1.7.0 - what approach would you recommend for renewing
tickets? Does it require stopping and starting the cluster?
Regards,
Roman
-----Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: 26 January 2016 18:10
To: user@accumulo.apache.org
Subject: Re: Accumulo and Kerberos
Hi Roman,
Accumulo services (TabletServer, Master, etc) all use a keytab to automatically
obtain a ticket from the KDC when they start up. You do not need to do anything
with kinit when starting Accumulo.
One worry is ACCUMULO-4069[1] with all presently released versions (most
notably 1.7.0 which you are using). This is a bug in which services did not
automatically renew their ticket. We're working on a 1.7.1, but it's not out
yet.
As for debugging your issue, take a look at the Kerberos section on debugging
in the user manual [2]. Take a very close look at the principal the service is
using to obtain the ticket and what the principal is for your keytab. A good
sanity check is to make sure you can `kinit` in the shell using the keytab and
the correct principal (rule out the keytab being incorrect).
If you still get stuck, collect the output specifying
-Dsun.security.krb5.debug=true in accumulo-env.sh (per the instructions) and
try enabling log4j DEBUG on org.apache.hadoop.security.UserGroupInformation.
- Josh
[1] https://issues.apache.org/jira/browse/ACCUMULO-4069
[2]
http://accumulo.apache.org/1.7/accumulo_user_manual.html#_debugging
roman.drap...@baesystems.com wrote:
Hi there,
Trying to setup Accumulo 1.7 on Kerberized cluster. Only interested
in master/tablets to be kerberized (not end-users). Configured
everything as per manual:
1)Created principals
2)Generated glob keytab
3)Modified accumulo-site.xml providing general.kerberos.keytab and
general.kerberos.principal
If I start as accumulo user I get: Caused by: GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
However, if I give explicitly a token with kinit and keytab
generated above in the shell - it works as expected. To my
understanding Accumulo has to obtain tickets automatically? Or the
idea is to write a cron job and apply kinit to every tablet server per day?
Regards,
Roman
Please consider the environment before printing this email. This
message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in
hard copy by an authorised signatory. The contents of this email may
relate to dealings with other companies under the control of BAE
Systems Applied Intelligence Limited, details of which can be found
at http://www.baesystems.com/Businesses/index.htm.
Please consider the environment before printing this email. This message should
be regarded as confidential. If you have received this email in error please
notify the sender and destroy it immediately. Statements of intent shall only
become binding when confirmed in hard copy by an authorised signatory. The
contents of this email may relate to dealings with other companies under the
control of BAE Systems Applied Intelligence Limited, details of which can be
found at http://www.baesystems.com/Businesses/index.htm.
Please consider the environment before printing this email. This message should
be regarded as confidential. If you have received this email in error please
notify the sender and destroy it immediately. Statements of intent shall only
become binding when confirmed in hard copy by an authorised signatory. The
contents of this email may relate to dealings with other companies under the
control of BAE Systems Applied Intelligence Limited, details of which can be
found at http://www.baesystems.com/Businesses/index.htm.
Please consider the environment before printing this email. This message should
be regarded as confidential. If you have received this email in error please
notify the sender and destroy it immediately. Statements of intent shall only
become binding when confirmed in hard copy by an authorised signatory. The
contents of this email may relate to dealings with other companies under the
control of BAE Systems Applied Intelligence Limited, details of which can be
found at http://www.baesystems.com/Businesses/index.htm.
Please consider the environment before printing this email. This message should
be regarded as confidential. If you have received this email in error please
notify the sender and destroy it immediately. Statements of intent shall only
become binding when confirmed in hard copy by an authorised signatory. The
contents of this email may relate to dealings with other companies under the
control of BAE Systems Applied Intelligence Limited, details of which can be
found at http://www.baesystems.com/Businesses/index.htm.
