[jira] [Commented] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297502#comment-15297502
]
Sergey Shelukhin commented on HIVE-13651:
-
[~jdere] can you comment wrt the above, and the compilation and materialization
logic for LLAPIF? What do we need to get the tokens during that compilation.
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297008#comment-15297008
]
Siddharth Seth commented on HIVE-13651:
---
For a regular Tez dag execution - HBase tokens are obtained specifically by the
Hive client. For HDFS tokens - hive provides a list of URIs to Tez, and Tez
takes care of obtaining the tokens.
The main scenario here will be access data on the local cluster. For HDFS -
this should be handled by LLAP dameons having access to data owned by Hive (no
tokens need to be propagated). I'm not sure how HBase will work - will the
kerberos credentials used by LLAP be sufficient to talk to HBase ?
For the non-standard case - e.g. 1. accessing data from a different cluster -
HS2 could obtain the tokens for the hive user. An important aspect to consider
here is whether HS2 will be able to perform security checks for an alternate
table. 2. Accessing data owned by some other user - this responsibility would
fall on the client (HS2 does not have the required credentials to do this).
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285635#comment-15285635
]
Sergey Shelukhin commented on HIVE-13651:
-
Actually, I think this is better done separately. LLAP token and non-LLAP
tokens come from different places
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13651) LlapBaseInputFormat: figure out where credentials come from
[
https://issues.apache.org/jira/browse/HIVE-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279151#comment-15279151
]
Sergey Shelukhin commented on HIVE-13651:
-
Oh, so this JIRA already exists. The serialized token will be a part of
getSplits UDF output. Does it make sense after that?
> LlapBaseInputFormat: figure out where credentials come from
> ---
>
> Key: HIVE-13651
> URL: https://issues.apache.org/jira/browse/HIVE-13651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>
> todo in LlapBaseInputFormat.constructSubmitWorkRequestProto()
> {code}
> // TODO Figure out where credentials will come from. Normally Hive sets up
> // URLs on the tez dag, for which Tez acquires credentials.
> //taskCredentials.addAll(getContext().getCredentials());
> //
> Preconditions.checkState(currentQueryIdentifierProto.getDagIdentifier() ==
> //
> taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId());
> //ByteBuffer credentialsBinary =
> credentialMap.get(currentQueryIdentifierProto);
> //if (credentialsBinary == null) {
> // credentialsBinary =
> serializeCredentials(getContext().getCredentials());
> // credentialMap.putIfAbsent(currentQueryIdentifierProto,
> credentialsBinary.duplicate());
> //} else {
> // credentialsBinary = credentialsBinary.duplicate();
> //}
> //
> builder.setCredentialsBinary(ByteString.copyFrom(credentialsBinary));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
