[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298816#comment-16298816
 ] 

Hudson commented on HELIX-599:
--

FAILURE: Integrated in Jenkins build helix #1383 (See 
[https://builds.apache.org/job/helix/1383/])
Revert "[HELIX-599] Support creating/maintaining/routing resources with (cji: 
rev 257f2dec2798f3806a97b28243d3f303372a2902)
* (edit) helix-core/src/main/java/org/apache/helix/model/Message.java
* (edit) helix-core/src/main/java/org/apache/helix/model/Resource.java
* (delete) 
helix-core/src/test/java/org/apache/helix/integration/TestResourceGroupEndtoEnd.java
* (edit) 
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
* (edit) 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java
* (edit) 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
* (edit) helix-core/src/main/java/org/apache/helix/model/IdealState.java
* (edit) helix-core/src/main/java/org/apache/helix/tools/ClusterSetup.java
* (edit) helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java
* (edit) helix-core/src/main/java/org/apache/helix/model/ExternalView.java
[HELIX-599] Support creating/maintaining/routing resources with same (jxue: rev 
63c4329aa0f3fa3a19a1080e40f0a000a49bcdcf)
* (edit) 
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
* (edit) 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java
* (edit) helix-core/src/main/java/org/apache/helix/model/IdealState.java
* (edit) helix-core/src/main/java/org/apache/helix/tools/ClusterSetup.java
* (edit) helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java
* (edit) helix-core/src/main/java/org/apache/helix/model/ExternalView.java
* (edit) helix-core/src/main/java/org/apache/helix/model/Message.java
* (edit) helix-core/src/main/java/org/apache/helix/model/Resource.java
* (add) 
helix-core/src/test/java/org/apache/helix/integration/TestResourceGroupEndtoEnd.java
* (edit) 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java


> Support creating/maintaining/routing resources with same names in different 
> instance groups
> ---
>
> Key: HELIX-599
> URL: https://issues.apache.org/jira/browse/HELIX-599
> Project: Apache Helix
>  Issue Type: New Feature
>  Components: helix-core, helix-webapp-admin
>Reporter: Lei Xia
>Assignee: Lei Xia
> Fix For: 0.6.5
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> In LinkedIn, we have a new use scenario that there will be multiple databases 
> sitting in the same Helix cluster with the same name, but on different 
> instance groups.  What we need are:
>  1) Allow resources (databases) with the same name, these resources are on 
> different instance groups (with different tags).
>  2) Routing table (Spectator) is able to aggregate and return all instance 
> (from multiple instance groups) that hold the database with given name.
> Our proposed solution is:
>  1) Add a "Resource Group" field in IdealState for the databases with the 
> same names from different instance groups
>  2) Use Instance Group Tag (or new "Resource Tag") to differentiate databases 
> (with same name) from different instance groups.
>  3) Use name mangling for Idealstate, for example, with database TestDB in 
> instance group "testGroup", the IdealState and ExternalView id would be 
> "TestDB$testGroup". 
>  4) Change Helix Routing Table to be able to aggregate databases from the 
> same resource group.
>  
> Four new APIs are going to be added to RoutingTableProvider:
> public class RoutingTableProvider {
>  
> /**
>  * returns the instances that contain the given partition in a specific state 
> from all resources with given resource name
>  */
> public List getInstances(String resource, String partition, 
> String state);
>  
> /**
>  * returns the instances that contain the given partition in a specific state 
> from selected resources with given name and tags
>  */
> public List getInstances(String resource, String partition, 
> String state, List resourceTags);
>  
> /**
>  * returns instances that contain given resource that are in a specific state
>  */
> public Set getInstances(String resource, String state);
>  
> /**
>  * returns instances that contain given resource with tags that are in a 
> specific state
>  */
> public Set getInstances(String resource, String state,  
> List groupTags);
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630371#comment-14630371
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user asfgit closed the pull request at:

https://github.com/apache/helix/pull/31


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621757#comment-14621757
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328333
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -73,6 +75,73 @@ public RoutingTableProvider() {
   }
 
   /**
+   * returns the instances for {resource,partition} pair that are in a 
specific {state} if
+   * aggregateGrouping is turned on, find all resources belongs to the 
given resourceGroupName and
+   * aggregate all partition states from all these resources.
+   *
+   * @param resourceName
+   * @param partitionName
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public ListInstanceConfig getInstances(String resourceName, String 
partitionName, String state,
--- End diff --

Yes, I kept this for back-compatible API.  I will change the new method to 
getInstancesForResourceGroup,  keep the old getInstances (and mark it as 
deprecated) and add another method getInstancesForResource.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621758#comment-14621758
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328348
  
--- Diff: helix-core/src/main/java/org/apache/helix/model/IdealState.java 
---
@@ -536,4 +574,16 @@ public boolean isEnabled() {
   public void enable(boolean enabled) {
 _record.setSimpleField(IdealStateProperty.HELIX_ENABLED.name(), 
Boolean.toString(enabled));
   }
+
+  /**
+   * Get the mangled IdealState name if resourceGroup is enable.
+   *
+   * @param resourceName
+   * @param resourceTag
+   *
+   * @return
+   */
+  public static String getIdealStateName(String resourceName, String 
resourceTag) {
--- End diff --

I will remove this method from IdealState, and put it into client side code 
(ClusterSetup).


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621770#comment-14621770
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328550
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
--- End diff --

move the comparator outside and defined it as a static variable in 
RoutingTableProvider class.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621771#comment-14621771
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328553
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
+
+  @Override
+  public int compare(InstanceConfig o1, InstanceConfig o2) {
+if (o1 == o2) {
+  return 0;
+}
+if (o1 == null) {
+  return -1;
+}
+if (o2 == null) {
+  return 1;
+}
+
+int compareTo = o1.getHostName().compareTo(o2.getHostName());
+if (compareTo == 0) {
+  return o1.getPort().compareTo(o2.getPort());
+} else {
+  return compareTo;
+}
+
+  }
+};
+stateInfoMap.put(state, new TreeSetInstanceConfig(comparator));
+  }
+  SetInstanceConfig set = stateInfoMap.get(state);
+  set.add(config);
+
+  if (!partitionInfoMap.containsKey(stateUnitKey)) {
+partitionInfoMap.put(stateUnitKey, new PartitionInfo());
+  }
+  PartitionInfo stateUnitKeyInfo = partitionInfoMap.get(stateUnitKey);
+  stateUnitKeyInfo.addEntry(state, config);
+
+  if (!tagResourceInfoMap.containsKey(resourceTag)) {
--- End diff --

It will not. 


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621772#comment-14621772
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328588
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
+
+  @Override
+  public int compare(InstanceConfig o1, InstanceConfig o2) {
+if (o1 == o2) {
+  return 0;
+}
+if (o1 == null) {
+  return -1;
+}
+if (o2 == null) {
+  return 1;
+}
+
+int compareTo = o1.getHostName().compareTo(o2.getHostName());
+if (compareTo == 0) {
+  return o1.getPort().compareTo(o2.getPort());
+} else {
+  return compareTo;
+}
+
+  }
+};
+stateInfoMap.put(state, new TreeSetInstanceConfig(comparator));
+  }
+  SetInstanceConfig set = stateInfoMap.get(state);
+  set.add(config);
+
+  if (!partitionInfoMap.containsKey(stateUnitKey)) {
+partitionInfoMap.put(stateUnitKey, new PartitionInfo());
+  }
+  PartitionInfo stateUnitKeyInfo = partitionInfoMap.get(stateUnitKey);
+  stateUnitKeyInfo.addEntry(state, config);
+
+  if (!tagResourceInfoMap.containsKey(resourceTag)) {
+tagResourceInfoMap.put(resourceTag, new ResourceInfo());
+  }
+  ResourceInfo resourceInfo = tagResourceInfoMap.get(resourceTag);
+  resourceInfo.addEntry(stateUnitKey, state, config);
+}
+
+public SetInstanceConfig getInstances(String state) {
+  SetInstanceConfig instanceSet = stateInfoMap.get(state);
+  return instanceSet;
+}
+
+public SetInstanceConfig getInstances(String state, String 
resourceTag) {
+  ResourceInfo resourceInfo = tagResourceInfoMap.get(resourceTag);
--- End diff --

Resources in the same group will never have the same tags, otherwise it 
will be rejected since their idealstate name will be same.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621773#comment-14621773
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328613
  
--- Diff: 
helix-core/src/test/java/org/apache/helix/integration/TestResourceGroupEndtoEnd.java
 ---
@@ -0,0 +1,465 @@
+package org.apache.helix.integration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import org.apache.helix.HelixAdmin;
+import org.apache.helix.HelixConstants;
+import org.apache.helix.TestHelper;
+import org.apache.helix.integration.manager.ClusterControllerManager;
+import org.apache.helix.integration.manager.MockParticipantManager;
+import org.apache.helix.integration.manager.ZkTestManager;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixAdmin;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.manager.zk.ZNRecordSerializer;
+import org.apache.helix.manager.zk.ZkClient;
+import org.apache.helix.mock.participant.DummyProcess;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.OnlineOfflineSMD;
+import org.apache.helix.spectator.RoutingTableProvider;
+import org.apache.helix.tools.ClusterSetup;
+import org.apache.helix.tools.ClusterStateVerifier;
+import org.testng.Assert;
+import org.apache.helix.HelixManager;
+import org.apache.helix.HelixManagerFactory;
+import org.apache.helix.InstanceType;
+import org.apache.helix.NotificationContext;
+import org.apache.helix.model.Message;
+import org.apache.helix.participant.StateMachineEngine;
+import org.apache.helix.participant.statemachine.StateModel;
+import org.apache.helix.participant.statemachine.StateModelFactory;
+import org.apache.log4j.Logger;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/*
--- End diff --

Fixed.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621747#comment-14621747
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328187
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
 ---
@@ -127,7 +127,9 @@ public void process(ClusterEvent event) throws 
Exception {
 Message message =
 createMessage(manager, resourceName, 
partition.getPartitionName(), instanceName,
 currentState, nextState, 
sessionIdMap.get(instanceName), stateModelDef.getId(),
-resource.getStateModelFactoryname(), bucketSize);
--- End diff --

Fix the method to take resource as argument for my updated pull request.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621748#comment-14621748
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328188
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
 ---
@@ -190,7 +192,8 @@ public void process(ClusterEvent event) throws 
Exception {
 
   private Message createMessage(HelixManager manager, String resourceName, 
String partitionName,
   String instanceName, String currentState, String nextState, String 
sessionId,
-  String stateModelDefName, String stateModelFactoryName, int 
bucketSize) {
--- End diff --

Fixed,


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621775#comment-14621775
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328709
  
--- Diff: helix-core/src/main/java/org/apache/helix/tools/ClusterSetup.java 
---
@@ -363,6 +373,43 @@ public void addResourceToCluster(String clusterName, 
String resourceName, int nu
 bucketSize, maxPartitionsPerInstance);
   }
 
+  /**
+   * Create an IdealState for a resource that belongs to a resource group 
We use
+   * resourceGroupName$resourceInstanceTag as the IdealState znode name 
to differetiate different
+   * resources from the same resourceGroup.
+   */
+  public IdealState createIdealStateForResourceGroup(String 
resourceGroupName,
+  String resourceTag, int numPartition, int replica, String 
rebalanceMode) {
+String idealStateId = IdealState.getIdealStateName(resourceGroupName, 
resourceTag);
+IdealState idealState = new IdealState(idealStateId);
+
+idealState.setNumPartitions(numPartition);
+idealState.setStateModelDefRef(OnlineOffline);
--- End diff --

Sorry, fixed.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621759#comment-14621759
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328375
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
 ---
@@ -62,6 +62,10 @@ public void process(ClusterEvent event) throws Exception 
{
   
resource.setStateModelFactoryName(idealState.getStateModelFactoryName());
   resource.setBucketSize(idealState.getBucketSize());
   resource.setBatchMessageMode(idealState.getBatchMessageMode());
+  if (idealState.isResourceGroupEnabled()) {
+
resource.setResourceGroupName(idealState.getResourceGroupName());
--- End diff --

Removed the checking, even if it is null, it is fine.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621761#comment-14621761
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328391
  
--- Diff: helix-core/src/main/java/org/apache/helix/model/ExternalView.java 
---
@@ -95,6 +105,32 @@ public String getResourceName() {
 return _record.getId();
   }
 
+  /**
+   * Get the resource group name
+   *
+   * @return the name of the resource group this resource belongs to.
+   */
+  public String getResourceGroupName() {
--- End diff --

RoutingTableProvider call this to determine whether it needs to cache all 
resource group partition/instance information.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621766#comment-14621766
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r34328491
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -91,6 +160,52 @@ public RoutingTableProvider() {
 return instanceSet;
   }
 
+  /**
+   * returns all instances for all resources from {resource} group that 
are in a specific {state}
+   *
+   * @param resource
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public SetInstanceConfig getInstances(String resource, String state, 
boolean groupingEnabled) {
+if (!groupingEnabled) {
+  return getInstances(resource, state);
+}
+
+SetInstanceConfig instanceSet = null;
+RoutingTable _routingTable = _routingTableRef.get();
+ResourceGroupInfo resourceGroupInfo = 
_routingTable.getResourceGroup(resource);
--- End diff --

fixed.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614141#comment-14614141
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892210
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
 ---
@@ -206,6 +209,13 @@ private Message createMessage(HelixManager manager, 
String resourceName, String
 message.setStateModelFactoryName(stateModelFactoryName);
 message.setBucketSize(bucketSize);
 
+if (resourceGroup != null) {
--- End diff --

Oh, I guess `ResourceComputationStage` does this


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614140#comment-14614140
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892206
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
 ---
@@ -206,6 +209,13 @@ private Message createMessage(HelixManager manager, 
String resourceName, String
 message.setStateModelFactoryName(stateModelFactoryName);
 message.setBucketSize(bucketSize);
 
+if (resourceGroup != null) {
--- End diff --

Should this also check `isResourceGroupEnabled`?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614142#comment-14614142
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892212
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java
 ---
@@ -62,6 +62,10 @@ public void process(ClusterEvent event) throws Exception 
{
   
resource.setStateModelFactoryName(idealState.getStateModelFactoryName());
   resource.setBucketSize(idealState.getBucketSize());
   resource.setBatchMessageMode(idealState.getBatchMessageMode());
+  if (idealState.isResourceGroupEnabled()) {
+
resource.setResourceGroupName(idealState.getResourceGroupName());
--- End diff --

Can this verify that resource group name and tag are non-null?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614165#comment-14614165
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892332
  
--- Diff: 
helix-core/src/test/java/org/apache/helix/integration/TestResourceGroupEndtoEnd.java
 ---
@@ -0,0 +1,465 @@
+package org.apache.helix.integration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import org.apache.helix.HelixAdmin;
+import org.apache.helix.HelixConstants;
+import org.apache.helix.TestHelper;
+import org.apache.helix.integration.manager.ClusterControllerManager;
+import org.apache.helix.integration.manager.MockParticipantManager;
+import org.apache.helix.integration.manager.ZkTestManager;
+import org.apache.helix.manager.zk.CallbackHandler;
+import org.apache.helix.manager.zk.ZKHelixAdmin;
+import org.apache.helix.manager.zk.ZKHelixManager;
+import org.apache.helix.manager.zk.ZNRecordSerializer;
+import org.apache.helix.manager.zk.ZkClient;
+import org.apache.helix.mock.participant.DummyProcess;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.OnlineOfflineSMD;
+import org.apache.helix.spectator.RoutingTableProvider;
+import org.apache.helix.tools.ClusterSetup;
+import org.apache.helix.tools.ClusterStateVerifier;
+import org.testng.Assert;
+import org.apache.helix.HelixManager;
+import org.apache.helix.HelixManagerFactory;
+import org.apache.helix.InstanceType;
+import org.apache.helix.NotificationContext;
+import org.apache.helix.model.Message;
+import org.apache.helix.participant.StateMachineEngine;
+import org.apache.helix.participant.statemachine.StateModel;
+import org.apache.helix.participant.statemachine.StateModelFactory;
+import org.apache.log4j.Logger;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+/*
--- End diff --

Move this notice above the imports, after the package declaration.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614143#comment-14614143
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892224
  
--- Diff: helix-core/src/main/java/org/apache/helix/model/ExternalView.java 
---
@@ -95,6 +105,32 @@ public String getResourceName() {
 return _record.getId();
   }
 
+  /**
+   * Get the resource group name
+   *
+   * @return the name of the resource group this resource belongs to.
+   */
+  public String getResourceGroupName() {
--- End diff --

Who calls this?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614156#comment-14614156
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892283
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -91,6 +160,52 @@ public RoutingTableProvider() {
 return instanceSet;
   }
 
+  /**
+   * returns all instances for all resources from {resource} group that 
are in a specific {state}
+   *
+   * @param resource
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public SetInstanceConfig getInstances(String resource, String state, 
boolean groupingEnabled) {
--- End diff --

Also, as @kishoreg mentioned above, it's better to have two methods rather 
than this boolean flag.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614155#comment-14614155
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892277
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -91,6 +160,52 @@ public RoutingTableProvider() {
 return instanceSet;
   }
 
+  /**
+   * returns all instances for all resources from {resource} group that 
are in a specific {state}
+   *
+   * @param resource
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public SetInstanceConfig getInstances(String resource, String state, 
boolean groupingEnabled) {
--- End diff --

Is it assumed that all resources in a given resource group have the same 
state machine?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614157#comment-14614157
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892286
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -91,6 +160,52 @@ public RoutingTableProvider() {
 return instanceSet;
   }
 
+  /**
+   * returns all instances for all resources from {resource} group that 
are in a specific {state}
+   *
+   * @param resource
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public SetInstanceConfig getInstances(String resource, String state, 
boolean groupingEnabled) {
+if (!groupingEnabled) {
+  return getInstances(resource, state);
+}
+
+SetInstanceConfig instanceSet = null;
+RoutingTable _routingTable = _routingTableRef.get();
+ResourceGroupInfo resourceGroupInfo = 
_routingTable.getResourceGroup(resource);
+if (resourceGroupInfo != null) {
+  instanceSet = resourceGroupInfo.getInstances(state);
+}
+if (instanceSet == null) {
+  instanceSet = Collections.emptySet();
+}
+return instanceSet;
+  }
+
+  public SetInstanceConfig getInstances(String resource, String state,
+  ListString resourceTags) {
+SetInstanceConfig instanceSet = null;
--- End diff --

Can you refactor these methods to increase the code reuse? They all seem to 
follow the same general pattern.


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614164#comment-14614164
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892316
  
--- Diff: helix-core/src/main/java/org/apache/helix/tools/ClusterSetup.java 
---
@@ -363,6 +373,43 @@ public void addResourceToCluster(String clusterName, 
String resourceName, int nu
 bucketSize, maxPartitionsPerInstance);
   }
 
+  /**
+   * Create an IdealState for a resource that belongs to a resource group 
We use
+   * resourceGroupName$resourceInstanceTag as the IdealState znode name 
to differetiate different
+   * resources from the same resourceGroup.
+   */
+  public IdealState createIdealStateForResourceGroup(String 
resourceGroupName,
+  String resourceTag, int numPartition, int replica, String 
rebalanceMode) {
+String idealStateId = IdealState.getIdealStateName(resourceGroupName, 
resourceTag);
+IdealState idealState = new IdealState(idealStateId);
+
+idealState.setNumPartitions(numPartition);
+idealState.setStateModelDefRef(OnlineOffline);
--- End diff --

Shouldn't this be an argument?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614163#comment-14614163
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892312
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
+
+  @Override
+  public int compare(InstanceConfig o1, InstanceConfig o2) {
+if (o1 == o2) {
+  return 0;
+}
+if (o1 == null) {
+  return -1;
+}
+if (o2 == null) {
+  return 1;
+}
+
+int compareTo = o1.getHostName().compareTo(o2.getHostName());
+if (compareTo == 0) {
+  return o1.getPort().compareTo(o2.getPort());
+} else {
+  return compareTo;
+}
+
+  }
+};
+stateInfoMap.put(state, new TreeSetInstanceConfig(comparator));
+  }
+  SetInstanceConfig set = stateInfoMap.get(state);
+  set.add(config);
+
+  if (!partitionInfoMap.containsKey(stateUnitKey)) {
+partitionInfoMap.put(stateUnitKey, new PartitionInfo());
+  }
+  PartitionInfo stateUnitKeyInfo = partitionInfoMap.get(stateUnitKey);
+  stateUnitKeyInfo.addEntry(state, config);
+
+  if (!tagResourceInfoMap.containsKey(resourceTag)) {
+tagResourceInfoMap.put(resourceTag, new ResourceInfo());
+  }
+  ResourceInfo resourceInfo = tagResourceInfoMap.get(resourceTag);
+  resourceInfo.addEntry(stateUnitKey, state, config);
+}
+
+public SetInstanceConfig getInstances(String state) {
+  SetInstanceConfig instanceSet = stateInfoMap.get(state);
+  return instanceSet;
+}
+
+public SetInstanceConfig getInstances(String state, String 
resourceTag) {
+  ResourceInfo resourceInfo = tagResourceInfoMap.get(resourceTag);
--- End diff --

What if multiple resources have the same tag?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614161#comment-14614161
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892305
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
+
+  @Override
+  public int compare(InstanceConfig o1, InstanceConfig o2) {
+if (o1 == o2) {
+  return 0;
+}
+if (o1 == null) {
+  return -1;
+}
+if (o2 == null) {
+  return 1;
+}
+
+int compareTo = o1.getHostName().compareTo(o2.getHostName());
+if (compareTo == 0) {
+  return o1.getPort().compareTo(o2.getPort());
+} else {
+  return compareTo;
+}
+
+  }
+};
+stateInfoMap.put(state, new TreeSetInstanceConfig(comparator));
+  }
+  SetInstanceConfig set = stateInfoMap.get(state);
+  set.add(config);
+
+  if (!partitionInfoMap.containsKey(stateUnitKey)) {
+partitionInfoMap.put(stateUnitKey, new PartitionInfo());
+  }
+  PartitionInfo stateUnitKeyInfo = partitionInfoMap.get(stateUnitKey);
+  stateUnitKeyInfo.addEntry(state, config);
+
+  if (!tagResourceInfoMap.containsKey(resourceTag)) {
--- End diff --

Can `resourceTag` be null?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614159#comment-14614159
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892301
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -235,8 +376,94 @@ PartitionInfo get(String stateUnitKey) {
 }
   }
 
+  class ResourceGroupInfo {
+// store PartitionInfo for each partition for all resources in the 
resource group
+HashMapString, PartitionInfo partitionInfoMap;
+// stores the Set of Instances in a given state for all resources in 
the resource group
+HashMapString, SetInstanceConfig stateInfoMap;
+// stores the ResourceInfo for each resource with different instance 
tag.
+HashMapString, ResourceInfo tagResourceInfoMap;
+
+public ResourceGroupInfo() {
+  partitionInfoMap = new HashMapString, 
RoutingTableProvider.PartitionInfo();
+  stateInfoMap = new HashMapString, SetInstanceConfig();
+  tagResourceInfoMap = new HashMapString, ResourceInfo();
+}
+
+public void addEntry(String resourceTag, String stateUnitKey, String 
state, InstanceConfig config) {
+  // add
+  if (!stateInfoMap.containsKey(state)) {
+ComparatorInstanceConfig comparator = new 
ComparatorInstanceConfig() {
--- End diff --

Declare the comparator outside since it seems to be identical to the one in 
`ResourceInfo`


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614150#comment-14614150
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892258
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -73,6 +75,73 @@ public RoutingTableProvider() {
   }
 
   /**
+   * returns the instances for {resource,partition} pair that are in a 
specific {state} if
+   * aggregateGrouping is turned on, find all resources belongs to the 
given resourceGroupName and
+   * aggregate all partition states from all these resources.
+   *
+   * @param resourceName
+   * @param partitionName
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public ListInstanceConfig getInstances(String resourceName, String 
partitionName, String state,
+  boolean groupingEnabled) {
+if (!groupingEnabled) {
+  return getInstances(resourceName, partitionName, state);
+}
+
+ListInstanceConfig instanceList = null;
+RoutingTable _routingTable = _routingTableRef.get();
+ResourceGroupInfo resourceGroupInfo = 
_routingTable.getResourceGroup(resourceName);
--- End diff --

`resourceName` should be named `resourceGroupName`, right?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614152#comment-14614152
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kanakb commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33892268
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/spectator/RoutingTableProvider.java 
---
@@ -73,6 +75,73 @@ public RoutingTableProvider() {
   }
 
   /**
+   * returns the instances for {resource,partition} pair that are in a 
specific {state} if
+   * aggregateGrouping is turned on, find all resources belongs to the 
given resourceGroupName and
+   * aggregate all partition states from all these resources.
+   *
+   * @param resourceName
+   * @param partitionName
+   * @param state
+   * @param groupingEnabled
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public ListInstanceConfig getInstances(String resourceName, String 
partitionName, String state,
+  boolean groupingEnabled) {
+if (!groupingEnabled) {
+  return getInstances(resourceName, partitionName, state);
+}
+
+ListInstanceConfig instanceList = null;
+RoutingTable _routingTable = _routingTableRef.get();
+ResourceGroupInfo resourceGroupInfo = 
_routingTable.getResourceGroup(resourceName);
+if (resourceGroupInfo != null) {
+  PartitionInfo keyInfo = resourceGroupInfo.get(partitionName);
+  if (keyInfo != null) {
+instanceList = keyInfo.get(state);
+  }
+}
+if (instanceList == null) {
+  instanceList = Collections.emptyList();
+}
+return instanceList;
+  }
+
+  /**
+   * returns the instances for {resource,partition} pair that are in a 
specific {state} if
+   * aggregateGrouping is turned on, find all resources belongs to the 
given resourceGroupName and
+   * having the given resource tags and return the aggregated partition 
states from all these
+   * resources.
+   *
+   * @param resourceGroupName
+   * @param partitionName
+   * @param state
+   * @param resourceTags
+   *
+   * @return empty list if there is no instance in a given state
+   */
+  public ListInstanceConfig getInstances(String resourceGroupName, 
String partitionName,
--- End diff --

Can you clarify the expected naming pattern for partitions in a resource 
group? Do all resources have partitions with the same name, or do some, or do 
none? Does this basically look at every resource in the resource group for 
partitions that match, and then return all instances for all partitions with 
this name in the resources for this resource group?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String 

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613486#comment-14613486
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kishoreg commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33882177
  
--- Diff: helix-core/src/main/java/org/apache/helix/model/IdealState.java 
---
@@ -536,4 +574,16 @@ public boolean isEnabled() {
   public void enable(boolean enabled) {
 _record.setSimpleField(IdealStateProperty.HELIX_ENABLED.name(), 
Boolean.toString(enabled));
   }
+
+  /**
+   * Get the mangled IdealState name if resourceGroup is enable.
+   *
+   * @param resourceName
+   * @param resourceTag
+   *
+   * @return
+   */
+  public static String getIdealStateName(String resourceName, String 
resourceTag) {
--- End diff --

why do we need this method? This convention can be completely handled on 
client side rt?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613461#comment-14613461
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kishoreg commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33881099
  
--- Diff: helix-core/src/main/java/org/apache/helix/model/IdealState.java 
---
@@ -55,7 +55,9 @@
 MAX_PARTITIONS_PER_INSTANCE,
 INSTANCE_GROUP_TAG,
 REBALANCER_CLASS_NAME,
-HELIX_ENABLED
+HELIX_ENABLED,
+RESOURCE_GROUP_NAME,
+RESOURCE_GROUP_ENABLED
--- End diff --

Why do we need ResourceGroupEnabled flag? Will things work as expected if 
there is a resourcegroupName and by default we can set the resourceGroupName to 
resourceName ?


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613460#comment-14613460
 ] 

ASF GitHub Bot commented on HELIX-599:
--

Github user kishoreg commented on a diff in the pull request:

https://github.com/apache/helix/pull/31#discussion_r33881054
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
 ---
@@ -190,7 +192,8 @@ public void process(ClusterEvent event) throws 
Exception {
 
   private Message createMessage(HelixManager manager, String resourceName, 
String partitionName,
   String instanceName, String currentState, String nextState, String 
sessionId,
-  String stateModelDefName, String stateModelFactoryName, int 
bucketSize) {
--- End diff --

This method has too many parameters, we need to just pass in resource or 
have a message builder class


 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-599) Support creating/maintaining/routing resources with same names in different instance groups

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596405#comment-14596405
 ] 

ASF GitHub Bot commented on HELIX-599:
--

GitHub user lei-xia opened a pull request:

https://github.com/apache/helix/pull/31

[HELIX-599] Support creating/maintaining/routing resources with same names 
in different instance groups.


More details on the problems and our proposed solution is on the jira 
description: https://issues.apache.org/jira/browse/HELIX-599

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lei-xia/helix helix-0.6.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/31.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #31


commit 2f88e070fb698c1420873c1bffa63640638de1ba
Author: Lei Xia l...@linkedin.com
Date:   2015-05-11T17:54:27Z

[HELIX-599] Support creating/maintaining/routing resources with same names 
in different instance groups.




 Support creating/maintaining/routing resources with same names in different 
 instance groups
 ---

 Key: HELIX-599
 URL: https://issues.apache.org/jira/browse/HELIX-599
 Project: Apache Helix
  Issue Type: New Feature
  Components: helix-core, helix-webapp-admin
Reporter: Lei Xia
Assignee: Lei Xia
   Original Estimate: 168h
  Remaining Estimate: 168h

 In LinkedIn, we have a new use scenario that there will be multiple databases 
 sitting in the same Helix cluster with the same name, but on different 
 instance groups.  What we need are:
  1) Allow resources (databases) with the same name, these resources are on 
 different instance groups (with different tags).
  2) Routing table (Spectator) is able to aggregate and return all instance 
 (from multiple instance groups) that hold the database with given name.
 Our proposed solution is:
  1) Add a Resource Group field in IdealState for the databases with the 
 same names from different instance groups
  2) Use Instance Group Tag (or new Resource Tag) to differentiate databases 
 (with same name) from different instance groups.
  3) Use name mangling for Idealstate, for example, with database TestDB in 
 instance group testGroup, the IdealState and ExternalView id would be 
 TestDB$testGroup. 
  4) Change Helix Routing Table to be able to aggregate databases from the 
 same resource group.
  
 Four new APIs are going to be added to RoutingTableProvider:
 public class RoutingTableProvider {
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from all resources with given resource name
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state);
  
 /**
  * returns the instances that contain the given partition in a specific state 
 from selected resources with given name and tags
  */
 public ListInstanceConfig getInstances(String resource, String partition, 
 String state, ListString resourceTags);
  
 /**
  * returns instances that contain given resource that are in a specific state
  */
 public SetInstanceConfig getInstances(String resource, String state);
  
 /**
  * returns instances that contain given resource with tags that are in a 
 specific state
  */
 public SetInstanceConfig getInstances(String resource, String state,  
 ListString groupTags);
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)