[
https://issues.apache.org/jira/browse/PIG-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niels Basjes updated PIG-4796:
------------------------------
Attachment: 2016-02-18-PIG-4796-rough-proof-of-concept.patch
Attaching the first *very rough proof of concept* version.
This seems to work on my machine most of the time.
Code is very rough around the edges so you may cut your fingers when trying
this.
The usage of this patch is by setting the variables as environment settings
like this:
{code}
#!/bin/bash
export krb5_conf="/etc/krb5.conf"
export krb5_principal="[email protected]"
export krb5_keytab="/home/nbasjes/.krb/nbasjes.keytab"
pig ./k.pig
{code}
I initially tried putting this in the ~/.pigbootup like this
{code}
SET java.security.krb5.conf '/etc/krb5.conf'
SET job.security.krb5.principal '[email protected]'
SET job.security.krb5.keytab '/home/nbasjes/.krb/nbasjes.keytab'
{code}
but right now it seems that this file is read AFTER the initial connect to the
secured HDFS is made (and thus this connection fails).
> Authenticate with Kerberos using a keytab file
> ----------------------------------------------
>
> Key: PIG-4796
> URL: https://issues.apache.org/jira/browse/PIG-4796
> Project: Pig
> Issue Type: New Feature
> Reporter: Niels Basjes
> Attachments: 2016-02-18-PIG-4796-rough-proof-of-concept.patch
>
>
> When running in a Kerberos secured environment users are faced with the
> limitation that their jobs cannot run longer than the (remaining) ticket
> lifetime of their Kerberos tickets. The environment I work in these tickets
> expire after 10 hours, thus limiting the maximum job duration to at most 10
> hours (which is a problem).
> In the Hadoop tooling there is a feature where you can authenticate using a
> Kerberos keytab file (essentially a file that contains the encrypted form of
> the kerberos principal and password). Using this the running application can
> request new tickets from the Kerberos server when the initial tickets expire.
> In my Java/Hadoop applications I commonly include these two lines:
> {code}
> System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");
> UserGroupInformation.loginUserFromKeytab("[email protected]",
> "/home/nbasjes/.krb/nbasjes.keytab");
> {code}
> This way I have run an Apache Flink based application for more than 170 hours
> (about a week) on the kerberos secured Yarn cluster.
> What I propose is to have a feature that I can set the relevant kerberos
> values in my pig script and from there be able to run a pig job for many days
> on the secured cluster.
> Proposal how this can look in a pig script:
> {code}
> SET java.security.krb5.conf '/etc/krb5.conf'
> SET job.security.krb5.principal '[email protected]'
> SET job.security.krb5.keytab '/home/nbasjes/.krb/nbasjes.keytab'
> {code}
> So iff all of these are set (or at least the last two) then the
> aforementioned UserGroupInformation.loginUserFromKeytab method is called
> before submitting the job to the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
