Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-15 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 7:03 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

minor update to avoid zk exceptions during shutdown. also unregister listener 
is removed as latch close will automatically remove listeners and no need to do 
explicit removal.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/6/

Changes: https://reviews.apache.org/r/65498/diff/5-6/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 15, 2018, 5:51 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

some changes to notification handling. added comments about how notifications 
are received and handled.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 907d064 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/5/

Changes: https://reviews.apache.org/r/65498/diff/4-5/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-14 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 14, 2018, 8:24 a.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Addressed review comments.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 30c24d9 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/4/

Changes: https://reviews.apache.org/r/65498/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 


> On March 13, 2018, 8:41 p.m., Sergey Shelukhin wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
> > Lines 123 (patched)
> > 
> >
> > nit: why is a thread needed when we just join immediately?

good catch.. join() supposed to happen together for the different threads.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 


> On March 13, 2018, 9:03 p.m., Sergey Shelukhin wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java
> > Lines 663 (patched)
> > 
> >
> > should this be addressed? the starting sessionstate will not be present 
> > iirc

Fixed.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199139
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


Moved the casts internally to getInstances()



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


Fixed.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


Good point. It is possible to lose leadership just before publishing it. 
Added a lock to protect around races. also reordered to move hasLeadership 
happen just before publishing endpoint.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


HIVE-18952 created



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


Sorry bad rebase. Fixed it.



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


Fixed it.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199136
---




llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


Fixed method signature. A path can have only one instance.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


UUID is static. This is for the unit test.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


these are defaults. llap and tez AM uses the defaults but HS2 will override 
or ignore some prefixes.. need c'tor override for all prefixes.

LLAP path:
/llap-sasl/hiveuser/hostname/workers/worker-000

HS2 path:
/hs2ActivePassiveHA-sasl/instances/instance-00



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


This is required for ZK based service discovery. Clients connecting using 
ZK based service discovery to HS2 will have namespace in the URL (cannot have 
subpaths like /ns/user/workers). 

JDBC URL will have hs2ActivePassiveHA as namespace.

For HS2 the ACL provider path will be
/hs2ActivePassiveHA-sasl/instances/

There is no per user restriction (JDBC URL doesn't get users).



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


IntelliJ rule violation :) final is unnecessary for static methods.



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


llap-common is not dependent on llap-client. Dropped..



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


ServiceRegistry interface had LlapServiceInstanceSet which is specific to 
LLAP. ServiceRegistry is updated to return generic ServiceInstanceSet. 
HiveServer2HAInstanceSet extends it to add isLeader() interface on top of 
ServiceRegistry.


- Prasanth_J


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199118
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 666 (patched)


JIRA? this seems pretty critical


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199114
---




service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 663 (patched)


should this be addressed? the starting sessionstate will not be present iirc



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 670 (patched)


the error handling for SSL tests has disappeared



service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 707 (patched)


there's a logic change here... WM should still be activated w/o resource 
plan if queue is present, so that activating a resource plan could enable it


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
>   
> service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65498/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199113
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java
Line 55 (original), 56 (patched)


same as other comment - why change many places to a more general type just 
to add casts in other places?
at least before the dependency on specific type was compile time verified, 
now it's run time verified



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 183 (patched)


until? should there be a followup jira? :)



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 198 (patched)


nit: maybe the latch listener can be hidden in a non-static nested class... 
otherwise the interface implementation is visible to the users and these 
methods are a bit confusing.



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 202 (patched)


just wondering, how is this protected from races?
is there some grace period? should this be configurable/documented?
I.e. is it possible to check hasLeadership() just before losing it, but 
still publish this?



service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
Lines 268 (patched)


nit: javadoc


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>   llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
> PRE-CREATION 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> 908b3bb 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
> 34fba5c 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  db3d788 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
> 0724cf5 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
>  417e571 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> 17269dd 
>   llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
> 9666517 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 0120639 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  58bf8dc 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  66de3b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 46cfe56 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a8d729d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> d261623 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java
>  PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
>  PRE-CREATION 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/#review199112
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 123 (patched)


nit: why is a thread needed when we just join immediately?



itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java
Lines 178 (patched)


nit: could it wait or loop on some condition for 3sec instead of sleeping? 
e.g. hs2Instances.size or leaders.size below



llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
Line 330 (original), 330 (patched)


hmm... why does getInstances now get one instance?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Lines 67 (patched)


why is this change necessary? shouldn't worker ID be enough?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
Line 65 (original), 73 (patched)


nit: this is missing another 31



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 84 (patched)


hmm... if these are moved here, should they still be parameters to ctor? I 
saw GROUP being added to all the ctors calling super(), but given that they are 
taken from here might as well just hide them inside here



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 86 (original), 91 (patched)


comment is out of date



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 168 (patched)


hmm.. I'm not sure what this change is about.
ACL provider may be called for multiple paths that do not exist; some may 
be top level paths that should have full access, and some should be user level 
paths that should be restricted.
Generally there's a clear boundary - everything above certain dir is open, 
everything at or below is restricted, so only one path argument is necessary.
Why is this change needed?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Line 406 (original), 459 (patched)


why not final?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 482 (patched)


should null be logged? not sure when it would be valid for correct data to 
have null here, if there's no bug of some sort



llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java
Lines 27 (patched)


hmm



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Line 1365 (original), 1375 (patched)


hmm... this cast doesn't look so good. Why was field type changed? Maybe 
the cast should be at assignment


- Sergey Shelukhin


On March 13, 2018, 6:41 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65498/
> ---
> 
> (Updated March 13, 2018, 6:41 p.m.)
> 
> 
> Review request for hive, Sergey Shelukhin and Thejas Nair.
> 
> 
> Bugs: HIVE-18281
> https://issues.apache.org/jira/browse/HIVE-18281
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18281: HiveServer2 HA for LLAP and Workload Manager
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
> PRE-CREATION 
>   
> itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java
>  6cab8cd 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5d7f813 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  c88198f 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  80a6aba 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  8339230 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  32d5caa 
>

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-03-13 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated March 13, 2018, 6:41 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Rebased patch


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aedd1ec 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
46cfe56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java bc438bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b7ece2b 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/3/

Changes: https://reviews.apache.org/r/65498/diff/2-3/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-02-05 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

(Updated Feb. 5, 2018, 11:05 p.m.)


Review request for hive, Sergey Shelukhin and Thejas Nair.


Changes
---

Added /peers endpoint to list all hs2 instances and their leadership status


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 99e8457 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cd 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 8bbf8a4 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f813 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198f 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6aba 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 8339230 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bb 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dd 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dc 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
82fdf6c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b98fb58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java b33f027 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 2a528cd 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2Peers.java PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/2/

Changes: https://reviews.apache.org/r/65498/diff/1-2/


Testing
---


Thanks,

Prasanth_J

Review Request 65498: HIVE-18281: HiveServer2 HA for LLAP and Workload Manager

2018-02-02 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65498/
---

Review request for hive, Sergey Shelukhin and Thejas Nair.


Bugs: HIVE-18281
https://issues.apache.org/jira/browse/HIVE-18281


Repository: hive-git


Description
---

HIVE-18281: HiveServer2 HA for LLAP and Workload Manager


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
4f2e6d31af85cf1d9866ad1419f5c06a18eea347 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestActivePassiveHA.java 
PRE-CREATION 
  
itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/AbstractHiveService.java 
6cab8cdd7d0f2dcc6c55800dba69786fad638cea 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
8bbf8a40e2400ea5ec56f76078b0e227dfe7e95a 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5d7f8131f54f7401888dae47f3b1e75ab3c81b06 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 c88198ffa154dd7ed405f7c79eca21a67eaba45a 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 80a6abaed4c015c43b08b610a42d6cb63fa678a8 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 83392309bb1dabfd4e93656385775b5bccb335d3 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
32d5caa51bec0c0ec97c641356278221280916d1 
  llap-client/src/java/org/apache/hadoop/hive/registry/RegistryUtilities.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
908b3bbe14196f86aaa090e0469a5c279ee55193 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstanceSet.java 
34fba5c3fab85073590260612cc59893c45da57d 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 db3d788de5def33b30023993820422a077e02c90 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmInstance.java 
0724cf5b1fc57101f79832def9dd2ff26f86643a 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/TezAmRegistryImpl.java
 417e571b8d884bc80edf9384d363ea6973cdaa39 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
17269dddf7ba8b43c34580311effcda5d3d27fbe 
  llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java 
9666517a354b97f02914f93773b22316ed872cf2 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
0120639955683b69886d262e07240b0526136d86 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 58bf8dce9c33dd000eb5c91975369d7abe21caed 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 66de3b805a752dd5527fd01e17939eb4b190f54f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
82fdf6cf2f0299ff78e63acad00b8b5e6b62ac90 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
b98fb58f0ba113f58ca0c724b3f3b0a58180c10d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 
b33f027e40481efd31989c9b18d94f5706138a63 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a8d729d797e0c616fde2cbaf9e7cc21895cc791f 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
d261623eb136b4f109210eb62ccb87d10e2174a8 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistry.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/server/HS2ActivePassiveHARegistryClient.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 
2a528cd14511d0db1df51b5785dc27425fa6f880 
  service/src/java/org/apache/hive/service/server/HiveServer2HAInstanceSet.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2Instance.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/servlet/HS2LeadershipStatus.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65498/diff/1/


Testing
---


Thanks,

Prasanth_J