[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[ https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895934#comment-16895934 ] Adam Szita commented on HIVE-21922: --- After consulting with other folks, it looks like this change is not desirable. In Hadoop world we're abusing Kerberos entities i.e. hive/host1@realm and hive/host2@realm are interpreted by UGI as the same Hive user. Still we need different principals per host so that LDAP doesn't revoke permissions due to frequent renewals seen across the cluster if one principal is used for Hive only. Thus marking this change as resolved. > Allow keytabs to be reused in LLAP yarn applications through Yarn localization > -- > > Key: HIVE-21922 > URL: https://issues.apache.org/jira/browse/HIVE-21922 > Project: Hive > Issue Type: New Feature >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch, > HIVE-21922.2.patch > > > In secure clusters LLAP has to be able to reach keytab files for kerberos > login. > Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and > _hive.llap.daemon.keytab.file_ configs are used to define the path of such > keytabs on the Tez AM and LLAP daemon side respectively. Both presume local > file system paths only - hence all nodes in the LLAP cluster (even those that > eventually don't end up executing a daemon...) have to have Hive's keytab > preinstalled on them. > The above is described by this strategy: > [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers] > Another approach can be > [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN] > where we rely on HDFS and Yarn resource localization, and no prior keytab > distribution is required. I intend to make this strategy an option for > Hive-LLAP in this jira. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873485#comment-16873485
]
Hive QA commented on HIVE-21922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17753/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17753/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17753/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972933 - PreCommit-HIVE-Build
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch,
> HIVE-21922.2.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
> preinstalled on them.
> The above is described by this strategy:
> [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers]
> Another approach can be
> [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN]
> where we rely on HDFS and Yarn resource localization, and no prior keytab
> distribution is required. I intend to make this strategy an option for
> Hive-LLAP in this jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873276#comment-16873276
]
Hive QA commented on HIVE-21922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972933/HIVE-21922.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16340 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17748/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17748/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972933 - PreCommit-HIVE-Build
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch,
> HIVE-21922.2.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
> preinstalled on them.
> The above is described by this strategy:
> [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers]
> Another approach can be
> [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN]
> where we rely on HDFS and Yarn resource localization, and no prior keytab
> distribution is required. I intend to make this strategy an option for
> Hive-LLAP in this jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873236#comment-16873236
]
Hive QA commented on HIVE-21922:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 8 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17748/dev-support/hive-personality.sh
|
| git revision | master / 967a1cc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17748/yetus/whitespace-tabs.txt
|
| modules | C: common ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17748/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch,
> HIVE-21922.2.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
>
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872985#comment-16872985
]
Hive QA commented on HIVE-21922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972894/HIVE-21922.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17744/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17744/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12972894/HIVE-21922.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972894 - PreCommit-HIVE-Build
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
> preinstalled on them.
> The above is described by this strategy:
> [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers]
> Another approach can be
> [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN]
> where we rely on HDFS and Yarn resource localization, and no prior keytab
> distribution is required. I intend to make this strategy an option for
> Hive-LLAP in this jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872695#comment-16872695
]
Hive QA commented on HIVE-21922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972894/HIVE-21922.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16307 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=232)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17733/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17733/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17733/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972894 - PreCommit-HIVE-Build
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
> preinstalled on them.
> The above is described by this strategy:
> [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers]
> Another approach can be
> [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN]
> where we rely on HDFS and Yarn resource localization, and no prior keytab
> distribution is required. I intend to make this strategy an option for
> Hive-LLAP in this jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[
https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872662#comment-16872662
]
Hive QA commented on HIVE-21922:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 8 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17733/dev-support/hive-personality.sh
|
| git revision | master / aed7500 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17733/yetus/whitespace-tabs.txt
|
| modules | C: common ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17733/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow keytabs to be reused in LLAP yarn applications through Yarn localization
> --
>
> Key: HIVE-21922
> URL: https://issues.apache.org/jira/browse/HIVE-21922
> Project: Hive
> Issue Type: New Feature
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch
>
>
> In secure clusters LLAP has to be able to reach keytab files for kerberos
> login.
> Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and
> _hive.llap.daemon.keytab.file_ configs are used to define the path of such
> keytabs on the Tez AM and LLAP daemon side respectively. Both presume local
> file system paths only - hence all nodes in the LLAP cluster (even those that
> eventually don't end up executing a daemon...) have to have Hive's keytab
> preinstalled on them.
> The above
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[ https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872539#comment-16872539 ] Adam Szita commented on HIVE-21922: --- Thanks [~pvary] I amended my patch with: * clearer documentation parts * in TezSessionState, I'm no longer writing keytab file path to this.conf, rather to tezConf. This is required, so that when opening a new tez session we will see "" for hive.llap.task.scheduler.am.registry.keytab.file if it was before.. > Allow keytabs to be reused in LLAP yarn applications through Yarn localization > -- > > Key: HIVE-21922 > URL: https://issues.apache.org/jira/browse/HIVE-21922 > Project: Hive > Issue Type: New Feature >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21922.0.patch, HIVE-21922.1.patch > > > In secure clusters LLAP has to be able to reach keytab files for kerberos > login. > Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and > _hive.llap.daemon.keytab.file_ configs are used to define the path of such > keytabs on the Tez AM and LLAP daemon side respectively. Both presume local > file system paths only - hence all nodes in the LLAP cluster (even those that > eventually don't end up executing a daemon...) have to have Hive's keytab > preinstalled on them. > The above is described by this strategy: > [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers] > Another approach can be > [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN] > where we rely on HDFS and Yarn resource localization, and no prior keytab > distribution is required. I intend to make this strategy an option for > Hive-LLAP in this jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[ https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872313#comment-16872313 ] Peter Vary commented on HIVE-21922: --- +1 pending tests. Do not forget to add the new config to the wiki: [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] > Allow keytabs to be reused in LLAP yarn applications through Yarn localization > -- > > Key: HIVE-21922 > URL: https://issues.apache.org/jira/browse/HIVE-21922 > Project: Hive > Issue Type: New Feature >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21922.0.patch > > > In secure clusters LLAP has to be able to reach keytab files for kerberos > login. > Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and > _hive.llap.daemon.keytab.file_ configs are used to define the path of such > keytabs on the Tez AM and LLAP daemon side respectively. Both presume local > file system paths only - hence all nodes in the LLAP cluster (even those that > eventually don't end up executing a daemon...) have to have Hive's keytab > preinstalled on them. > The above is described by this strategy: > [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers] > Another approach can be > [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN] > where we rely on HDFS and Yarn resource localization, and no prior keytab > distribution is required. I intend to make this strategy an option for > Hive-LLAP in this jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21922) Allow keytabs to be reused in LLAP yarn applications through Yarn localization
[ https://issues.apache.org/jira/browse/HIVE-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872206#comment-16872206 ] Adam Szita commented on HIVE-21922: --- The patch introduces the following new options: * In Hive conf ** *hive.llap.use.hs2.keytab.for.am.registry.keytab*: if set to true and hive.llap.task.scheduler.am.registry.keytab.file is empty, HS2 keytab will be added to Yarn as resource to be localized for Tez AM use * In LLAP's yarn service descriptor file compiler python script: ** *service-keytab-localized-path*: if set, Yarn will make sure LLAP daemons can reach the keytab file on this path, earlier uploaded to HDFS path as per service-keytab-dir / service-keytab options [~pvary] can you take a look please? > Allow keytabs to be reused in LLAP yarn applications through Yarn localization > -- > > Key: HIVE-21922 > URL: https://issues.apache.org/jira/browse/HIVE-21922 > Project: Hive > Issue Type: New Feature >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21922.0.patch > > > In secure clusters LLAP has to be able to reach keytab files for kerberos > login. > Currently _hive.llap.task.scheduler.am.registry.keytab.file_ and > _hive.llap.daemon.keytab.file_ configs are used to define the path of such > keytabs on the Tez AM and LLAP daemon side respectively. Both presume local > file system paths only - hence all nodes in the LLAP cluster (even those that > eventually don't end up executing a daemon...) have to have Hive's keytab > preinstalled on them. > The above is described by this strategy: > [Pre-installed_Keytabs_for_AM_and_containers|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Pre-installed_Keytabs_for_AM_and_containers] > Another approach can be > [Keytabs_for_AM_and_containers_distributed_via_YARN|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Keytabs_for_AM_and_containers_distributed_via_YARN] > where we rely on HDFS and Yarn resource localization, and no prior keytab > distribution is required. I intend to make this strategy an option for > Hive-LLAP in this jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
