subject:"Re\: Review Request 40315\: HIVE\-12341 LLAP security"

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109414
---

Ship it!


Looks good. Small nits in the last patch - mostly around creating new jiras.
The compiled proto file seems to be missing from the RB patch. Assuming that'll 
be part of the commit.

- Siddharth Seth


On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Dec. 5, 2015, 9:40 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java
>  f116de4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  a085427 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
>  2673ad7 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> dbdf571 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  b93650d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  8144165 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 52ba360 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  bf8a673 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 7c38dc3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109413
---

Ship it!


Ship It!

- Siddharth Seth


On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Dec. 5, 2015, 9:40 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java
>  f116de4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  a085427 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
>  2673ad7 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> dbdf571 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  b93650d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  8144165 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 52ba360 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  bf8a673 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 7c38dc3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109407
---



llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java 
(line 47)


rename to  getManagementPort ?



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 (line 156)


Not for this jira. While the daemons serve the management protocol - the 
management protocol and daemon protocol could potentially run on the same 
server with the same port. This saveson the number of handler threads which end 
up being created.



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 (line 159)


In the interim - a separate configuration for management handlers would 
help, which could be set to a low values - maybe 2.



llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 (line 66)


New jira to make this configurable


- Siddharth Seth


On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Dec. 5, 2015, 9:40 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java
>  f116de4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  a085427 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
>  2673ad7 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> dbdf571 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  b93650d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  8144165 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 52ba360 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  bf8a673 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 7c38dc3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-05 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Dec. 5, 2015, 9:40 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java 
f116de4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 a085427 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
 2673ad7 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
dbdf571 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 b93650d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 8144165 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
52ba360 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 bf8a673 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-03 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Dec. 3, 2015, 11:07 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f96071 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java 
f116de4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 34e0682 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java
 d474b6f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
4525ab9 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-01 Thread Lefty Leverenz


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108467
---

Ship it!


Ship It!

- Lefty Leverenz


On Nov. 30, 2015, 7:50 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 30, 2015, 7:50 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 4525ab9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-01 Thread Lefty Leverenz



> On Nov. 30, 2015, 7:49 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 2377-2378
> > 
> >
> > Should this have a TimeValidator (like the next two Spark parameters, 
> > lines 2372-2376)?
> > 
> > Also, the default is 14 * 24 * 3600 but the comment says 
> > DelegationTokenManager default is 1 week instead of 2 weeks, so is 
> > DelegationTokenManager something different from this parameter?
> 
> Sergey Shelukhin wrote:
> Yes, that is the hadoop delegation token manager. This is a comment for 
> the background of how the ballpark for the default value was chosen.

Okay, thanks Sergey.


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
---


On Nov. 30, 2015, 7:50 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 30, 2015, 7:50 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 4525ab9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-30 Thread Sergey Shelukhin



> On Nov. 30, 2015, 7:49 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 2377-2378
> > 
> >
> > Should this have a TimeValidator (like the next two Spark parameters, 
> > lines 2372-2376)?
> > 
> > Also, the default is 14 * 24 * 3600 but the comment says 
> > DelegationTokenManager default is 1 week instead of 2 weeks, so is 
> > DelegationTokenManager something different from this parameter?

Yes, that is the hadoop delegation token manager. This is a comment for the 
background of how the ballpark for the default value was chosen.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
---


On Nov. 26, 2015, 1:38 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 26, 2015, 1:38 a.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 4525ab9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-30 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 30, 2015, 7:50 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
4525ab9 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-29 Thread Lefty Leverenz


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2357 - 2358)


Spell out ZooKeeper in the description.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2359 - 2361)


Spell out ZooKeeper in the description.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2362 - 2363)


Spell out ZooKeeper in the description.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2367 - 2368)


Should this have a TimeValidator (like the next two Spark parameters, lines 
2372-2376)?

Also, the default is 14 * 24 * 3600 but the comment says 
DelegationTokenManager default is 1 week instead of 2 weeks, so is 
DelegationTokenManager something different from this parameter?


- Lefty Leverenz


On Nov. 26, 2015, 1:38 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 26, 2015, 1:38 a.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
> PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
> 4525ab9 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-25 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 26, 2015, 1:38 a.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 
4525ab9 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-25 Thread Sergey Shelukhin



> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > 
> >
> > Don't think the default value - "*" - has any significance here. 
> > Replace by null - to avoid confusion.
> 
> Sergey Shelukhin wrote:
> Following in the footsteps of slider
> 
> Siddharth Seth wrote:
> Can we drop the * in favor of a null. (or no default if Hive supports 
> that).

Why?


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java,
> >  line 129
> > 
> >
> > This could be moved into it's own protocol (but listening on the same 
> > server).
> > 
> > The methods so far are for access from the AM.
> > 
> > getTokens is to be used by Clients.
> > 
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a 
> > token.
> > Remaining methods - require a token.
> 
> Sergey Shelukhin wrote:
> why add an extra protocol? it seems like most services don't handle 
> tokens like this. HDFS gives out tokens as part of normal interface.
> 
> Siddharth Seth wrote:
> HDFS also doesn't have a separation of API vs server side jars.
> 
> Reason for separation is that they're very different operations - one 
> relates to exuecuting and tracking work, the other to access. Consider the 
> case where there is a central service which is responsible for handing out 
> these delegation tokens (rotation, etc etc). That will definitely not 
> implement the submitWork APIs, and the daemons cannot implement the getTokns 
> at that point.
> 
> Doesn't need to be done rightnow, and can be changed later since this API 
> is private for the moment (It is private, right?)
> 
> Re-opened primarily for the "Is this private" bit.

Added new protocol. Will test after the weekend.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
---


On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 24, 2015, 11:11 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apac

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth



> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > 
> >
> > Don't think the default value - "*" - has any significance here. 
> > Replace by null - to avoid confusion.
> 
> Sergey Shelukhin wrote:
> Following in the footsteps of slider

Can we drop the * in favor of a null. (or no default if Hive supports that).


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java,
> >  line 71
> > 
> >
> > Does a renewer for a token type have to be specified ?
> 
> Sergey Shelukhin wrote:
> renewer is set elsewhere

Not the user which does the renewal. Implementation of the Renewer interface. 
See TrivialRenewer in Token.java.
Not sure if one is required for each token kind - would be safer to add one 
which accepts the LLAP token kind and says it's not managed for now.


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java,
> >  line 129
> > 
> >
> > This could be moved into it's own protocol (but listening on the same 
> > server).
> > 
> > The methods so far are for access from the AM.
> > 
> > getTokens is to be used by Clients.
> > 
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a 
> > token.
> > Remaining methods - require a token.
> 
> Sergey Shelukhin wrote:
> why add an extra protocol? it seems like most services don't handle 
> tokens like this. HDFS gives out tokens as part of normal interface.

HDFS also doesn't have a separation of API vs server side jars.

Reason for separation is that they're very different operations - one relates 
to exuecuting and tracking work, the other to access. Consider the case where 
there is a central service which is responsible for handing out these 
delegation tokens (rotation, etc etc). That will definitely not implement the 
submitWork APIs, and the daemons cannot implement the getTokns at that point.

Doesn't need to be done rightnow, and can be changed later since this API is 
private for the moment (It is private, right?)

Re-opened primarily for the "Is this private" bit.


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java,
> >  line 251
> > 
> >
> > YARN can take care of renewing delegation tokens - assuming the service 
> > supports it (i.e. the ZKSecretManager on one of the LLAP instances or a 
> > direct connection to ZK from the RM - but that isn't a good idea).
> > Eventually, I believe the renweer would need to change to the RM 
> > service user.
> 
> Sergey Shelukhin wrote:
> Hmm... not sure how this would work. Can you file a follow-up JIRA? By MR 
> logic, the renewer would be a central job manager, e.g. HS2

YARN (The RM) is given tokens for a job. It checks whether these tokens are 
managed or not (via the Renewer interface), and takes care of renewing them 
while the job is alive. That's what is done for HDFS delegation tokens anyway - 
renewed by the RM every day while the job is running.
The same could be done for LLAP - but this is for later.
Opening a jira to track renewal of tokens.


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java,
> >  line 26
> > 
> >
> > How is the default value picked up ? (definitely not from the hive conf)
> > OR
> > What is the default value - "*" or " ".
> > I'm not sure how other services handle this - but  this can be set to " 
> > " by default on secure clusters, and "*" on non-secure clusters.
> 
> Sergey Shelukhin wrote:
> From the conf passed to refreshServiceAcl it looks like.

What if the fiels is not set ?


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java,
> >  line 32
> > 
> >
> > clone not required.
> 
> Sergey Shelukhin wrote:
> that looks like what other services do

Looking at yarn services, this isn't required. A final array is being returned 
- this was likely changed to get past findbugs warnings, and ignored in YARN


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/Llap

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107877
---


- Siddharth Seth


On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 24, 2015, 11:11 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107870
---


Token and credentials usage looks good to me. Looking at other review comments 
in a bit.


llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 (line 100)


Precondition check for security==true and token!=null


- Siddharth Seth


On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 24, 2015, 11:11 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 
> 59ee347 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 914b4e7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> efcf88c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 24, 2015, 11:11 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 24, 2015, 11:08 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Changes
---

same as previous minus the generated code


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 24, 2015, 11:06 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 b044df9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
914b4e7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
efcf88c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Sergey Shelukhin



> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > 
> >
> > Don't think the default value - "*" - has any significance here. 
> > Replace by null - to avoid confusion.

Following in the footsteps of slider


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java,
> >  line 71
> > 
> >
> > Does a renewer for a token type have to be specified ?

renewer is set elsewhere


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java,
> >  line 129
> > 
> >
> > This could be moved into it's own protocol (but listening on the same 
> > server).
> > 
> > The methods so far are for access from the AM.
> > 
> > getTokens is to be used by Clients.
> > 
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a 
> > token.
> > Remaining methods - require a token.

why add an extra protocol? it seems like most services don't handle tokens like 
this. HDFS gives out tokens as part of normal interface.


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java,
> >  line 134
> > 
> >
> > Sanity checks for the values. Empty strings are not allowed.

these are passed as defaults to ZK config getters; then, they are checked when 
attempting to log in


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java,
> >  line 251
> > 
> >
> > YARN can take care of renewing delegation tokens - assuming the service 
> > supports it (i.e. the ZKSecretManager on one of the LLAP instances or a 
> > direct connection to ZK from the RM - but that isn't a good idea).
> > Eventually, I believe the renweer would need to change to the RM 
> > service user.

Hmm... not sure how this would work. Can you file a follow-up JIRA? By MR 
logic, the renewer would be a central job manager, e.g. HS2


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java,
> >  line 26
> > 
> >
> > How is the default value picked up ? (definitely not from the hive conf)
> > OR
> > What is the default value - "*" or " ".
> > I'm not sure how other services handle this - but  this can be set to " 
> > " by default on secure clusters, and "*" on non-secure clusters.

>From the conf passed to refreshServiceAcl it looks like.


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java,
> >  line 32
> > 
> >
> > clone not required.

that looks like what other services do


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java,
> >  line 53
> > 
> >
> > This would matter when running under HiveServer ? or is the 
> > synchronization in LlapIoProxy taking care of this ?

shouldn't matter


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java,
> >  line 105
> > 
> >
> > Stop logging the token.

This part doesn't contain any secrets, changed to debug


> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java,
> >  line 511
> > 
> >
> > Required for each host separately ? Setting the host may not be 
> > required.

I think it is. Server has to run the principal with _HOST for hadoop IPC to work


On Nov. 24, 2015, 3:56 a.m., Sergey Shelukhin wrote:
> > Haven't looked at the details of the ZKSecretManager - but it looks like 
> > the Tokens issued by any of the LLAP instances can be used by an 
> > application to communicate with all other instances.
> > Also, are the tokens the same for different applications ?

Yes,

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-23 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
---

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2348)

Description needs fixing.

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2351)

Don't think the default value - "*" - has any significance here. Replace by
null - to avoid confusion.

llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java (line
76)

Rename class to LLAPProxy ? It's no longer limited to IO only.

llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
(line 44)

This and readFields - eventually should be implemented using a Protobuf
payload. Allows the token to evolve during rolling upgrades.
Separate jira.

llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
(line 71)

Does a renewer for a token type have to be specified ?

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
(line 128)

This could be moved into it's own protocol (but listening on the same
server).

The methods so far are for access from the AM.

getTokens is to be used by Clients.

What that also allows is for the annotations to change.
getTokens() - protected by Kerberos, and cannot be obtained using a token.
Remaining methods - require a token.

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
(line 129)

Rename to getDelegationToken ?

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
(line 79)

throws IOException not required.

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
(line 133)

Sanity checks for the values. Empty strings are not allowed.

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
(line 157)

Avoid using the ZK property names. Instead, the properties defined for LLAP
should be used.
(ZK properties could leak in from some other service which uses the same
SecretManager)

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
(line 164)

New instances of Configuration if doing a set (this config is shared by
several services)

llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
(line 250)

YARN can take care of renewing delegation tokens - assuming the service
supports it (i.e. the ZKSecretManager on one of the LLAP instances or a direct
connection to ZK from the RM - but that isn't a good idea).
Eventually, I believe the renweer would need to change to the RM service
user.

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
(line 26)

How is the default value picked up ? (definitely not from the hive conf)
OR
What is the default value - "*" or " ".
I'm not sure how other services handle this - but this can be set to " "
by default on secure clusters, and "*" on non-secure clusters.

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
(line 32)

clone not required.

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
(line 53)

This would matter when running under HiveServer ? or is the synchronization
in LlapIoProxy taking care of this ?

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
(line 59)

final. Also retryPolicy.

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
(line 67)

make configurable ?

llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
(line 94)

Lots of TODOs - fix / convert to jira with a ref

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-23 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107670
---



llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 (line 31)


TODO: need to clone, same below



llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 (line 96)


TODO debug


- Sergey Shelukhin


On Nov. 16, 2015, 7:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> ---
> 
> (Updated Nov. 16, 2015, 7:45 p.m.)
> 
> 
> Review request for hive, Gopal V and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
> 4c31e32 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
>  5ad2344 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
> 98b1ccd 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
>  4b13277 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
>  784c631 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
>  fae7654 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  d327fc0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
>  PRE-CREATION 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
>  33e998c 
>   
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
>  PRE-CREATION 
>   llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
>  8d45c95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
>   serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 
> 
> Diff: https://reviews.apache.org/r/40315/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-16 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
---

(Updated Nov. 16, 2015, 7:45 p.m.)


Review request for hive, Gopal V and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 
4c31e32 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java
 5ad2344 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
98b1ccd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java
 4b13277 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java
 784c631 
  
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java
 fae7654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 d327fc0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java
 33e998c 
  
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java
 8d45c95 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4 

Diff: https://reviews.apache.org/r/40315/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

Re: Review Request 40315: HIVE-12341 LLAP security

22 matches

Site Navigation

Mail list logo

Footer information