[jira] [Created] (HELIX-779) Maintenance rebalancer should not clear preference list in ideal state
Harry Zhang created HELIX-779: - Summary: Maintenance rebalancer should not clear preference list in ideal state Key: HELIX-779 URL: https://issues.apache.org/jira/browse/HELIX-779 Project: Apache Helix Issue Type: Bug Components: helix-core Reporter: Harry Zhang Assignee: Harry Zhang Setting list fields to empty map will prevent newly added and initially rebalanced resources during maintenance mode from getting re-balanced after cluster exists maintenance mode. The right thing to do is to clear every preference list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-779) Maintenance rebalancer should not clear preference list in ideal state
[ https://issues.apache.org/jira/browse/HELIX-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671943#comment-16671943 ] ASF GitHub Bot commented on HELIX-779: -- GitHub user zhan849 opened a pull request: https://github.com/apache/helix/pull/285 [HELIX-779] do not clean list field in maintenance rebalancer for new resources Setting list fields to empty map will prevent newly added and initially rebalanced resources during maintenance mode from getting re-balanced after cluster exists maintenance mode. The right thing to do is to clear every preference list. Also added test case to verify You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhan849/helix harry/maintenance-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #285 commit bfaa8399529b6e63b307c1fbe60903c3ca08fbb1 Author: Harry Zhang Date: 2018-10-04T22:50:16Z [HELIX-779] do not clean list field in maintenance rebalancer for new resources > Maintenance rebalancer should not clear preference list in ideal state > -- > > Key: HELIX-779 > URL: https://issues.apache.org/jira/browse/HELIX-779 > Project: Apache Helix > Issue Type: Bug > Components: helix-core >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Setting list fields to empty map will prevent newly added and initially > rebalanced resources during maintenance mode from getting re-balanced after > cluster exists maintenance mode. > The right thing to do is to clear every preference list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HELIX-780) Support get/add rest api for workflow/job/task user content
Harry Zhang created HELIX-780: - Summary: Support get/add rest api for workflow/job/task user content Key: HELIX-780 URL: https://issues.apache.org/jira/browse/HELIX-780 Project: Apache Helix Issue Type: Task Reporter: Harry Zhang Assignee: Harry Zhang Need to support get/add rest api for workflow/job/task user content AC: * finish implementation * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
helix git commit: [HELIX-779] do not clean list field in maintenance rebalancer for new resources
Repository: helix
Updated Branches:
refs/heads/master 89f351558 -> bfaa83995
[HELIX-779] do not clean list field in maintenance rebalancer for new resources
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/bfaa8399
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/bfaa8399
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/bfaa8399
Branch: refs/heads/master
Commit: bfaa8399529b6e63b307c1fbe60903c3ca08fbb1
Parents: 89f3515
Author: Harry Zhang
Authored: Thu Oct 4 15:50:16 2018 -0700
Committer: Harry Zhang
Committed: Thu Nov 1 10:57:44 2018 -0700
--
.../rebalancer/MaintenanceRebalancer.java | 7 ++-
.../controller/TestClusterMaintenanceMode.java | 22
2 files changed, 24 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/bfaa8399/helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java
b/helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java
index 3de3a3d..8bcf999 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java
@@ -24,7 +24,12 @@ public class MaintenanceRebalancer extends
SemiAutoRebalancer {
if (currentStateMap == null || currentStateMap.size() == 0) {
LOG.warn(String
.format("No new partition will be assigned for %s in maintenance
mode", resourceName));
- currentIdealState.setPreferenceLists(Collections.EMPTY_MAP);
+
+ // Clear all preference lists, if the resource has not yet been
rebalanced,
+ // leave it as is
+ for (List pList :
currentIdealState.getPreferenceLists().values()) {
+pList.clear();
+ }
return currentIdealState;
}
http://git-wip-us.apache.org/repos/asf/helix/blob/bfaa8399/helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java
b/helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java
index 4e4771b..2eb8034 100644
---
a/helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java
+++
b/helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java
@@ -13,6 +13,8 @@ import org.testng.annotations.Test;
public class TestClusterMaintenanceMode extends TaskTestBase {
MockParticipantManager _newInstance;
+ private String newResourceAddedDuringMaintenanceMode =
+ String.format("%s_%s", WorkflowGenerator.DEFAULT_TGT_DB, 1);
@BeforeClass
public void beforeClass() throws Exception {
@@ -53,13 +55,13 @@ public class TestClusterMaintenanceMode extends
TaskTestBase {
@Test (dependsOnMethods = "testMaintenanceModeAddNewInstance")
public void testMaintenanceModeAddNewResource() throws InterruptedException {
_gSetupTool.getClusterManagementTool()
-.addResource(CLUSTER_NAME, WorkflowGenerator.DEFAULT_TGT_DB + 1, 7,
"MasterSlave",
+.addResource(CLUSTER_NAME, newResourceAddedDuringMaintenanceMode, 7,
"MasterSlave",
IdealState.RebalanceMode.FULL_AUTO.name());
_gSetupTool.getClusterManagementTool()
-.rebalance(CLUSTER_NAME, WorkflowGenerator.DEFAULT_TGT_DB + 1, 3);
+.rebalance(CLUSTER_NAME, newResourceAddedDuringMaintenanceMode, 3);
Assert.assertTrue(_clusterVerifier.verifyByPolling());
ExternalView externalView = _gSetupTool.getClusterManagementTool()
-.getResourceExternalView(CLUSTER_NAME,
WorkflowGenerator.DEFAULT_TGT_DB + 1);
+.getResourceExternalView(CLUSTER_NAME,
newResourceAddedDuringMaintenanceMode);
Assert.assertNull(externalView);
}
@@ -88,4 +90,16 @@ public class TestClusterMaintenanceMode extends TaskTestBase
{
}
}
}
-}
\ No newline at end of file
+
+ @Test (dependsOnMethods = "testMaintenanceModeInstanceBack")
+ public void testExitMaintenanceModeNewResourceRecovery() throws
InterruptedException {
+_gSetupTool.getClusterManagementTool().enableMaintenanceMode(CLUSTER_NAME,
false);
+Assert.assertTrue(_clusterVerifier.verifyByPolling());
+ExternalView externalView = _gSetupTool.getClusterManagementTool()
+.getResourceExternalView(CLUSTER_NAME,
newResourceAddedDuringMaintenanceMode);
+Assert.assertEquals(externalView.getRecord().getMapFields().size(),
[jira] [Commented] (HELIX-779) Maintenance rebalancer should not clear preference list in ideal state
[ https://issues.apache.org/jira/browse/HELIX-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671989#comment-16671989 ] ASF GitHub Bot commented on HELIX-779: -- Github user asfgit closed the pull request at: https://github.com/apache/helix/pull/285 > Maintenance rebalancer should not clear preference list in ideal state > -- > > Key: HELIX-779 > URL: https://issues.apache.org/jira/browse/HELIX-779 > Project: Apache Helix > Issue Type: Bug > Components: helix-core >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Setting list fields to empty map will prevent newly added and initially > rebalanced resources during maintenance mode from getting re-balanced after > cluster exists maintenance mode. > The right thing to do is to clear every preference list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-779) Maintenance rebalancer should not clear preference list in ideal state
[ https://issues.apache.org/jira/browse/HELIX-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671995#comment-16671995 ] Hudson commented on HELIX-779: -- FAILURE: Integrated in Jenkins build helix #1562 (See [https://builds.apache.org/job/helix/1562/]) [HELIX-779] do not clean list field in maintenance rebalancer for new (hrzhang: rev bfaa8399529b6e63b307c1fbe60903c3ca08fbb1) * (edit) helix-core/src/test/java/org/apache/helix/integration/controller/TestClusterMaintenanceMode.java * (edit) helix-core/src/main/java/org/apache/helix/controller/rebalancer/MaintenanceRebalancer.java > Maintenance rebalancer should not clear preference list in ideal state > -- > > Key: HELIX-779 > URL: https://issues.apache.org/jira/browse/HELIX-779 > Project: Apache Helix > Issue Type: Bug > Components: helix-core >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Setting list fields to empty map will prevent newly added and initially > rebalanced resources during maintenance mode from getting re-balanced after > cluster exists maintenance mode. > The right thing to do is to clear every preference list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
helix git commit: [HELIX-780] add get/add user content for workflow rest api
Repository: helix
Updated Branches:
refs/heads/master bfaa83995 -> 71e4b6a66
[HELIX-780] add get/add user content for workflow rest api
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/71e4b6a6
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/71e4b6a6
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/71e4b6a6
Branch: refs/heads/master
Commit: 71e4b6a66af1ae56a3667d5f6f5ca7ac63080997
Parents: bfaa839
Author: Harry Zhang
Authored: Tue Oct 2 18:01:30 2018 -0700
Committer: Harry Zhang
Committed: Thu Nov 1 11:57:03 2018 -0700
--
.../resources/helix/WorkflowAccessor.java | 66 +++-
.../helix/rest/server/AbstractTestClass.java| 23 +++
.../helix/rest/server/TestWorkflowAccessor.java | 38 +++
3 files changed, 124 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/71e4b6a6/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java
--
diff --git
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java
index 9a9a62b..ac6a53c 100644
---
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java
+++
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java
@@ -21,6 +21,7 @@ package org.apache.helix.rest.server.resources.helix;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
@@ -34,24 +35,26 @@ import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Response;
-
+import org.I0Itec.zkclient.exception.ZkNoNodeException;
import org.apache.helix.HelixException;
import org.apache.helix.ZNRecord;
import org.apache.helix.task.JobConfig;
import org.apache.helix.task.JobDag;
import org.apache.helix.task.JobQueue;
import org.apache.helix.task.TaskDriver;
+import org.apache.helix.task.UserContentStore;
import org.apache.helix.task.Workflow;
import org.apache.helix.task.WorkflowConfig;
import org.apache.helix.task.WorkflowContext;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.type.TypeFactory;
import org.codehaus.jackson.node.ArrayNode;
import org.codehaus.jackson.node.JsonNodeFactory;
import org.codehaus.jackson.node.ObjectNode;
import org.codehaus.jackson.node.TextNode;
+import org.codehaus.jackson.type.TypeReference;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
@Path("/clusters/{clusterId}/workflows")
public class WorkflowAccessor extends AbstractHelixResource {
@@ -263,6 +266,63 @@ public class WorkflowAccessor extends
AbstractHelixResource {
}
@GET
+ @Path("{workflowId}/userContent")
+ public Response getWorkflowUserContent(
+ @PathParam("clusterId") String clusterId,
+ @PathParam("workflowId") String workflowId
+ ) {
+TaskDriver taskDriver = getTaskDriver(clusterId);
+try {
+ Map contentStore =
+ taskDriver.getWorkflowUserContentMap(workflowId);
+ if (contentStore == null) {
+return JSONRepresentation(Collections.emptyMap());
+ }
+ return JSONRepresentation(contentStore);
+} catch (ZkNoNodeException e) {
+ return notFound("Unable to find content store");
+} catch (Exception e) {
+ return serverError(e);
+}
+ }
+
+ @POST
+ @Path("{workflowId}/userContent")
+ public Response updateWorkflowUserContent(
+ @PathParam("clusterId") String clusterId,
+ @PathParam("workflowId") String workflowId,
+ @QueryParam("command") String commandStr,
+ String content
+ ) {
+Command cmd;
+Map contentMap = Collections.emptyMap();
+try {
+ contentMap = OBJECT_MAPPER.readValue(content, new
TypeReference>() {});
+ cmd = Command.valueOf(commandStr);
+} catch (IOException e) {
+ return badRequest(String.format("Content %s cannot be deserialized to
Map. Err: %s", content, e.getMessage()));
+} catch (IllegalArgumentException ie) {
+ return badRequest(String.format("Invalid command: %s. Err: %s",
commandStr, ie.getMessage()));
+} catch (NullPointerException npe) {
+ cmd = Command.update;
+}
+
+TaskDriver driver = getTaskDriver(clusterId);
+try {
+ switch (cmd) {
+ case update:
+driver.addOrUpdateWorkflowUserContentMap(workflowId, contentMap);
+return OK();
+ default:
+return badRequest(String.format("Command \"%s\" is not supported!",
cmd));
+ }
+
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672032#comment-16672032 ] ASF GitHub Bot commented on HELIX-780: -- Github user asfgit closed the pull request at: https://github.com/apache/helix/pull/286 > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672042#comment-16672042 ] Hudson commented on HELIX-780: -- FAILURE: Integrated in Jenkins build helix #1563 (See [https://builds.apache.org/job/helix/1563/]) [HELIX-780] add get/add user content for workflow rest api (hrzhang: rev 71e4b6a66af1ae56a3667d5f6f5ca7ac63080997) * (edit) helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/TestWorkflowAccessor.java > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672053#comment-16672053 ] ASF GitHub Bot commented on HELIX-780: -- GitHub user zhan849 opened a pull request: https://github.com/apache/helix/pull/287 [HELIX-780] add get/add job user content rest api added apis and tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhan849/helix harry/tf-rest-api Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #287 commit a09a18ac55464c3e399800b4474ccb6e64d168ec Author: Harry Zhang Date: 2018-10-08T22:36:53Z [HELIX-780] add get/add job user content rest api > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
helix git commit: [HELIX-780] add get/add job user content rest api
Repository: helix
Updated Branches:
refs/heads/master 71e4b6a66 -> a09a18ac5
[HELIX-780] add get/add job user content rest api
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/a09a18ac
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/a09a18ac
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/a09a18ac
Branch: refs/heads/master
Commit: a09a18ac55464c3e399800b4474ccb6e64d168ec
Parents: 71e4b6a
Author: Harry Zhang
Authored: Mon Oct 8 15:36:53 2018 -0700
Committer: Harry Zhang
Committed: Thu Nov 1 12:10:05 2018 -0700
--
.../server/resources/helix/JobAccessor.java | 68 ++--
.../helix/rest/server/TestJobAccessor.java | 38 +++
2 files changed, 102 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/a09a18ac/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
--
diff --git
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
index 9a085f1..a984428 100644
---
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
+++
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
@@ -19,7 +19,6 @@ package org.apache.helix.rest.server.resources.helix;
* under the License.
*/
-
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
@@ -28,12 +27,13 @@ import java.util.Set;
import javax.ws.rs.DELETE;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.GET;
+import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Response;
-
+import org.I0Itec.zkclient.exception.ZkNoNodeException;
import org.apache.helix.HelixException;
import org.apache.helix.ZNRecord;
import org.apache.helix.task.JobConfig;
@@ -41,11 +41,12 @@ import org.apache.helix.task.JobContext;
import org.apache.helix.task.TaskConfig;
import org.apache.helix.task.TaskDriver;
import org.apache.helix.task.WorkflowConfig;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import org.codehaus.jackson.node.ArrayNode;
import org.codehaus.jackson.node.JsonNodeFactory;
import org.codehaus.jackson.node.ObjectNode;
+import org.codehaus.jackson.type.TypeReference;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
@Path("/clusters/{clusterId}/workflows/{workflowName}/jobs")
public class JobAccessor extends AbstractHelixResource {
@@ -171,6 +172,65 @@ public class JobAccessor extends AbstractHelixResource {
return badRequest("Job context for " + jobName + " does not exists");
}
+ @GET
+ @Path("{jobName}/userContent")
+ public Response getJobUserContent(@PathParam("clusterId") String clusterId,
+ @PathParam("workflowName") String workflowName, @PathParam("jobName")
String jobName) {
+TaskDriver taskDriver = getTaskDriver(clusterId);
+try {
+ Map contentStore =
+ taskDriver.getJobUserContentMap(workflowName, jobName);
+ if (contentStore == null) {
+return JSONRepresentation(Collections.emptyMap());
+ }
+ return JSONRepresentation(contentStore);
+} catch (ZkNoNodeException e) {
+ return notFound("Unable to find content store");
+} catch (Exception e) {
+ return serverError(e);
+}
+ }
+
+ @POST
+ @Path("{jobName}/userContent")
+ public Response updateWorkflowUserContent(
+ @PathParam("clusterId") String clusterId,
+ @PathParam("workflowName") String workflowName,
+ @PathParam("jobName") String jobName,
+ @QueryParam("command") String commandStr,
+ String content
+ ) {
+Command cmd;
+Map contentMap = Collections.emptyMap();
+try {
+ contentMap = OBJECT_MAPPER.readValue(content, new
TypeReference>() {
+ });
+ cmd = (commandStr == null || commandStr.isEmpty())
+ ? Command.update
+ : Command.valueOf(commandStr);
+} catch (IOException e) {
+ return badRequest(String
+ .format("Content %s cannot be deserialized to Map.
Err: %s", content,
+ e.getMessage()));
+} catch (IllegalArgumentException ie) {
+ return badRequest(String.format("Invalid command: %s. Err: %s",
commandStr, ie.getMessage()));
+}
+
+TaskDriver driver = getTaskDriver(clusterId);
+try {
+ switch (cmd) {
+ case update:
+driver.addOrUpdateJobUserContentMap(workflowName, jobName, contentMap);
+return OK();
+ default:
+return badRequest(String.format("Command \"%s\" is not supported!",
cmd));
+ }
+} catch (Exce
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672227#comment-16672227 ] ASF GitHub Bot commented on HELIX-780: -- Github user asfgit closed the pull request at: https://github.com/apache/helix/pull/287 > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672230#comment-16672230 ] Hudson commented on HELIX-780: -- FAILURE: Integrated in Jenkins build helix #1564 (See [https://builds.apache.org/job/helix/1564/]) [HELIX-780] add get/add job user content rest api (hrzhang: rev a09a18ac55464c3e399800b4474ccb6e64d168ec) * (edit) helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/TestJobAccessor.java > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HELIX-781) Implement Helix cluster view aggregator
Harry Zhang created HELIX-781: - Summary: Implement Helix cluster view aggregator Key: HELIX-781 URL: https://issues.apache.org/jira/browse/HELIX-781 Project: Apache Helix Issue Type: Task Reporter: Harry Zhang Assignee: Harry Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[6/8] helix git commit: Fix test TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit
Fix test TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit
The test becomes unstable due to pipeline runtime changed.
Change to poll and wait style check.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/c783ae78
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/c783ae78
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/c783ae78
Branch: refs/heads/master
Commit: c783ae788f03c55122858eb552963de5c9a264dd
Parents: 30c0eff
Author: Jiajun Wang
Authored: Thu Oct 11 17:37:15 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:48 2018 -0700
--
...ceModeWhenReachingOfflineInstancesLimit.java | 31 +++-
1 file changed, 17 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/c783ae78/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit.java
b/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit.java
index 0de2fe3..9067bb5 100644
---
a/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit.java
+++
b/helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit.java
@@ -19,7 +19,6 @@ package org.apache.helix.integration.rebalancer;
* under the License.
*/
-import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.util.ArrayList;
import java.util.Date;
@@ -30,6 +29,7 @@ import org.apache.helix.ConfigAccessor;
import org.apache.helix.HelixAdmin;
import org.apache.helix.HelixDataAccessor;
import org.apache.helix.PropertyKey;
+import org.apache.helix.TestHelper;
import org.apache.helix.common.ZkTestBase;
import org.apache.helix.integration.manager.ClusterControllerManager;
import org.apache.helix.integration.manager.MockParticipantManager;
@@ -197,19 +197,22 @@ public class
TestClusterInMaintenanceModeWhenReachingOfflineInstancesLimit
System.out.println("END " + CLASS_NAME + " at " + new
Date(System.currentTimeMillis()));
}
- private void checkForRebalanceError(boolean expectError)
- throws MalformedObjectNameException, AttributeNotFoundException,
MBeanException,
- ReflectionException, InstanceNotFoundException, IOException {
-/* TODO re-enable this check when we start recording rebalance error again
-ZKHelixDataAccessor accessor = new ZKHelixDataAccessor(CLUSTER_NAME,
_baseAccessor);
-PropertyKey errorNodeKey =
-
accessor.keyBuilder().controllerTaskError(RebalanceResourceFailure.name());
-Assert.assertEquals(accessor.getProperty(errorNodeKey) != null,
expectError);
-*/
-
-Long value =
-(Long) _server.getAttribute(getClusterMbeanName(CLUSTER_NAME),
"RebalanceFailureGauge");
-Assert.assertEquals(value != null && value.longValue() > 0, expectError);
+ private void checkForRebalanceError(final boolean expectError) throws
Exception {
+boolean result = TestHelper.verify(new TestHelper.Verifier() {
+ @Override
+ public boolean verify() throws Exception {
+/* TODO re-enable this check when we start recording rebalance error
again
+ZKHelixDataAccessor accessor = new ZKHelixDataAccessor(CLUSTER_NAME,
_baseAccessor);
+PropertyKey errorNodeKey =
+
accessor.keyBuilder().controllerTaskError(RebalanceResourceFailure.name());
+Assert.assertEquals(accessor.getProperty(errorNodeKey) != null,
expectError);
+*/
+Long value =
+(Long) _server.getAttribute(getClusterMbeanName(CLUSTER_NAME),
"RebalanceFailureGauge");
+return expectError == (value != null && value.longValue() > 0);
+ }
+}, 5000);
+Assert.assertTrue(result);
}
private void cleanupRebalanceError() {
[4/8] helix git commit: Emitting per resource rebalance status for possible calculation failure.
Emitting per resource rebalance status for possible calculation failure.
The status in MBean will be string for debug purposes only.
The resource rebalance state attribute will be in one of the following state:
1. NORMAL
2. BEST_POSSIBLE_STATE_CAL_FAILED: calculation failed or no possible allocation
found.
3. INTERMEDIATE_STATE_CAL_FAILED: Intermediate state calculation failed. (not
include throttled case)
4. UNKNOWN: the resource is not rebalanced or newly created.
Additional related changes:
1. Fix a cluster level metric related bug to generate the right metrics data.
2. Fix a resource monitoring bug that DISABLE_MONITORING is not working.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/2f39f381
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/2f39f381
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/2f39f381
Branch: refs/heads/master
Commit: 2f39f381b0981503d7c204aabbeaa09153292e15
Parents: b549cda
Author: Jiajun Wang
Authored: Fri Oct 5 16:26:11 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:36 2018 -0700
--
.../stages/BestPossibleStateCalcStage.java | 52
.../stages/ExternalViewComputeStage.java| 4 +-
.../stages/IntermediateStateCalcStage.java | 19 +++
.../monitoring/mbeans/ClusterStatusMonitor.java | 66 ++
.../monitoring/mbeans/ResourceMonitor.java | 35 --
.../dynamicMBeans/DynamicMBeanProvider.java | 18 +--
.../TestAlertingRebalancerFailure.java | 123 +--
...ceModeWhenReachingOfflineInstancesLimit.java | 61 ++---
.../mbeans/TestClusterStatusMonitor.java| 10 +-
.../mbeans/TestDisableResourceMbean.java| 17 ++-
.../monitoring/mbeans/TestResourceMonitor.java | 21 +++-
11 files changed, 291 insertions(+), 135 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/2f39f381/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java
b/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java
index 1bbd6a0..b0e453d 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java
@@ -43,6 +43,7 @@ import org.apache.helix.model.Resource;
import org.apache.helix.model.ResourceAssignment;
import org.apache.helix.model.StateModelDefinition;
import org.apache.helix.monitoring.mbeans.ClusterStatusMonitor;
+import org.apache.helix.monitoring.mbeans.ResourceMonitor;
import org.apache.helix.task.TaskConstants;
import org.apache.helix.task.TaskRebalancer;
import org.apache.helix.util.HelixUtil;
@@ -75,11 +76,6 @@ public class BestPossibleStateCalcStage extends
AbstractBaseStage {
// Reset current INIT/RUNNING tasks on participants for throttling
cache.resetActiveTaskCount(currentStateOutput);
-// Check whether the offline/disabled instance count in the cluster
reaches the set limit,
-// if yes, pause the rebalancer.
-validateOfflineInstancesLimit(cache,
-(HelixManager) event.getAttribute(AttributeName.helixmanager.name()),
clusterStatusMonitor);
-
final BestPossibleStateOutput bestPossibleStateOutput =
compute(event, resourceMap, currentStateOutput);
event.addAttribute(AttributeName.BEST_POSSIBLE_STATE.name(),
bestPossibleStateOutput);
@@ -112,6 +108,13 @@ public class BestPossibleStateCalcStage extends
AbstractBaseStage {
BestPossibleStateOutput output = new BestPossibleStateOutput();
HelixManager helixManager =
event.getAttribute(AttributeName.helixmanager.name());
+ClusterStatusMonitor clusterStatusMonitor =
+event.getAttribute(AttributeName.clusterStatusMonitor.name());
+
+// Check whether the offline/disabled instance count in the cluster
reaches the set limit,
+// if yes, pause the rebalancer.
+boolean isValid = validateOfflineInstancesLimit(cache,
+(HelixManager) event.getAttribute(AttributeName.helixmanager.name()));
final List failureResources = new ArrayList<>();
Iterator itr = resourceMap.values().iterator();
@@ -125,6 +128,7 @@ public class BestPossibleStateCalcStage extends
AbstractBaseStage {
LogUtil.logError(logger, _eventId,
"Exception when calculating best possible states for " +
resource.getResourceName(),
ex);
+
}
if (!result) {
failureResources.add(resource.getResourceName());
@@ -134,31 +138,34 @@ public class BestPossibleStateCalcStage extends
AbstractBaseStage {
}
[2/8] helix git commit: Fix TestBucketizedResource.
Fix TestBucketizedResource.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/f9bc9f8c
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/f9bc9f8c
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/f9bc9f8c
Branch: refs/heads/master
Commit: f9bc9f8c69a480c87f89435b8b59f3ca04e8aeb0
Parents: 3721c1f
Author: Lei Xia
Authored: Wed Oct 10 10:14:25 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:27 2018 -0700
--
.../helix/integration/TestBucketizedResource.java | 16
1 file changed, 12 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/f9bc9f8c/helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java
b/helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java
index 27a7729..d10234d 100644
---
a/helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java
+++
b/helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java
@@ -121,7 +121,7 @@ public class TestBucketizedResource extends ZkTestBase {
}
@Test
- public void testBounceDisableAndDrop() {
+ public void testBounceDisableAndDrop() throws Exception {
String className = TestHelper.getTestClassName();
String methodName = TestHelper.getTestMethodName();
String clusterName = className + "_" + methodName;
@@ -154,7 +154,7 @@ public class TestBucketizedResource extends ZkTestBase {
participants[0].syncStop();
participants[0] = new MockParticipantManager(ZK_ADDR, clusterName,
instanceNames.get(0));
participants[0].syncStart();
-
+
Assert.assertTrue(_clusterVerifier.verifyByPolling());
// make sure participants[0]'s current state is bucketzied correctly
during carryover
@@ -176,8 +176,16 @@ public class TestBucketizedResource extends ZkTestBase {
Assert.assertTrue(_clusterVerifier.verifyByPolling());
// make sure external-view is cleaned up
-path = keyBuilder.externalView(dbName).getPath();
-boolean result = _baseAccessor.exists(path, 0);
+final String evPath = keyBuilder.externalView(dbName).getPath();
+
+TestHelper.verify(new TestHelper.Verifier() {
+@Override
+public boolean verify() {
+ return !_baseAccessor.exists(evPath, 0);
+}
+ }, 3000);
+
+boolean result = _baseAccessor.exists(evPath, 0);
Assert.assertFalse(result);
// clean up
[3/8] helix git commit: Change all Helix default created ZkClients to use ZnRecordSerializer.
Change all Helix default created ZkClients to use ZnRecordSerializer.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/b549cda9
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/b549cda9
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/b549cda9
Branch: refs/heads/master
Commit: b549cda95cb114da78efc4b0458058862bcc6d02
Parents: f9bc9f8
Author: Lei Xia
Authored: Wed Sep 19 16:34:46 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:31 2018 -0700
--
.../apache/helix/manager/zk/ZKHelixManager.java | 2 +-
.../helix/manager/zk/ZNRecordSerializer.java| 17 +--
.../manager/zk/TestZNRecordSerializer.java | 151 +++
.../zk/TestZNRecordStreamingSerializer.java | 19 +++
4 files changed, 177 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/b549cda9/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java
b/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java
index c673f51..c4275df 100644
--- a/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java
+++ b/helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixManager.java
@@ -594,7 +594,7 @@ public class ZKHelixManager implements HelixManager,
IZkStateListener {
void createClient() throws Exception {
PathBasedZkSerializer zkSerializer =
-ChainedPathZkSerializer.builder(new
ZNRecordStreamingSerializer()).build();
+ChainedPathZkSerializer.builder(new ZNRecordSerializer()).build();
HelixZkClient.ZkConnectionConfig connectionConfig = new
HelixZkClient.ZkConnectionConfig(_zkAddress);
connectionConfig.setSessionTimeout(_sessionTimeout);
http://git-wip-us.apache.org/repos/asf/helix/blob/b549cda9/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
b/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
index 95ebc06..0c92224 100644
---
a/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
+++
b/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
@@ -21,12 +21,8 @@ package org.apache.helix.manager.zk;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.io.StringWriter;
import java.util.List;
import java.util.Map;
-import java.util.zip.GZIPInputStream;
-import java.util.zip.GZIPOutputStream;
import org.I0Itec.zkclient.serialize.ZkSerializer;
import org.apache.helix.HelixException;
@@ -40,6 +36,7 @@ import org.codehaus.jackson.map.SerializationConfig;
public class ZNRecordSerializer implements ZkSerializer {
private static Logger logger =
LoggerFactory.getLogger(ZNRecordSerializer.class);
+ private final ObjectMapper _mapper = new ObjectMapper();
private static int getListFieldBound(ZNRecord record) {
int max = Integer.MAX_VALUE;
@@ -78,15 +75,14 @@ public class ZNRecordSerializer implements ZkSerializer {
}
// do serialization
-ObjectMapper mapper = new ObjectMapper();
-SerializationConfig serializationConfig = mapper.getSerializationConfig();
+SerializationConfig serializationConfig = _mapper.getSerializationConfig();
serializationConfig.set(SerializationConfig.Feature.INDENT_OUTPUT, true);
serializationConfig.set(SerializationConfig.Feature.AUTO_DETECT_FIELDS,
true);
serializationConfig.set(SerializationConfig.Feature.CAN_OVERRIDE_ACCESS_MODIFIERS,
true);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
-byte[] serializedBytes = null;
+byte[] serializedBytes;
try {
- mapper.writeValue(baos, data);
+ _mapper.writeValue(baos, data);
serializedBytes = baos.toByteArray();
// apply compression if needed
if (record.getBooleanField("enableCompression", false) ||
serializedBytes.length > ZNRecord.SIZE_LIMIT) {
@@ -113,10 +109,9 @@ public class ZNRecordSerializer implements ZkSerializer {
return null;
}
-ObjectMapper mapper = new ObjectMapper();
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
-DeserializationConfig deserializationConfig =
mapper.getDeserializationConfig();
+DeserializationConfig deserializationConfig =
_mapper.getDeserializationConfig();
deserializationConfig.set(DeserializationConfig.Feature.AUTO_DETECT_FIELDS,
true);
deserializationConfig.set(DeserializationConfig.Feature.AUTO_DETECT_SETTERS,
true);
[5/8] helix git commit: Revert the ObjectMapper change in ZKRecordSerializer.
Revert the ObjectMapper change in ZKRecordSerializer.
We noticed OOM in the test after this change. And verified that this is causing
the problem, revert it for now to unblocking test.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/30c0eff3
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/30c0eff3
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/30c0eff3
Branch: refs/heads/master
Commit: 30c0eff3ea04cabb6e3c3fe161fe457100f1868e
Parents: 2f39f38
Author: Jiajun Wang
Authored: Thu Oct 11 16:58:36 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:43 2018 -0700
--
.../org/apache/helix/manager/zk/ZNRecordSerializer.java | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/30c0eff3/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
b/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
index 0c92224..890bb13 100644
---
a/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
+++
b/helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java
@@ -36,7 +36,6 @@ import org.codehaus.jackson.map.SerializationConfig;
public class ZNRecordSerializer implements ZkSerializer {
private static Logger logger =
LoggerFactory.getLogger(ZNRecordSerializer.class);
- private final ObjectMapper _mapper = new ObjectMapper();
private static int getListFieldBound(ZNRecord record) {
int max = Integer.MAX_VALUE;
@@ -75,14 +74,15 @@ public class ZNRecordSerializer implements ZkSerializer {
}
// do serialization
-SerializationConfig serializationConfig = _mapper.getSerializationConfig();
+ObjectMapper mapper = new ObjectMapper();
+SerializationConfig serializationConfig = mapper.getSerializationConfig();
serializationConfig.set(SerializationConfig.Feature.INDENT_OUTPUT, true);
serializationConfig.set(SerializationConfig.Feature.AUTO_DETECT_FIELDS,
true);
serializationConfig.set(SerializationConfig.Feature.CAN_OVERRIDE_ACCESS_MODIFIERS,
true);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] serializedBytes;
try {
- _mapper.writeValue(baos, data);
+ mapper.writeValue(baos, data);
serializedBytes = baos.toByteArray();
// apply compression if needed
if (record.getBooleanField("enableCompression", false) ||
serializedBytes.length > ZNRecord.SIZE_LIMIT) {
@@ -111,7 +111,8 @@ public class ZNRecordSerializer implements ZkSerializer {
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
-DeserializationConfig deserializationConfig =
_mapper.getDeserializationConfig();
+ObjectMapper mapper = new ObjectMapper();
+DeserializationConfig deserializationConfig =
mapper.getDeserializationConfig();
deserializationConfig.set(DeserializationConfig.Feature.AUTO_DETECT_FIELDS,
true);
deserializationConfig.set(DeserializationConfig.Feature.AUTO_DETECT_SETTERS,
true);
deserializationConfig.set(DeserializationConfig.Feature.FAIL_ON_UNKNOWN_PROPERTIES,
true);
@@ -121,7 +122,7 @@ public class ZNRecordSerializer implements ZkSerializer {
byte[] uncompressedBytes = GZipCompressionUtil.uncompress(bais);
bais = new ByteArrayInputStream(uncompressedBytes);
}
- ZNRecord zn = _mapper.readValue(bais, ZNRecord.class);
+ ZNRecord zn = mapper.readValue(bais, ZNRecord.class);
return zn;
} catch (Exception e) {
[1/8] helix git commit: Allow to stop periodical rebalancer by resetting the clusterconfig. Add additional log for debug.
Repository: helix
Updated Branches:
refs/heads/master a09a18ac5 -> 44d7d2eb8
Allow to stop periodical rebalancer by resetting the clusterconfig. Add
additional log for debug.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/3721c1fb
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/3721c1fb
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/3721c1fb
Branch: refs/heads/master
Commit: 3721c1fbcfbb3ea40be7d485f341b013753210c6
Parents: a09a18a
Author: Lei Xia
Authored: Wed Oct 3 19:07:30 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:21 2018 -0700
--
.../java/org/apache/helix/HelixProperty.java| 8 ++-
.../helix/common/caches/AbstractDataCache.java | 7 ++
.../controller/GenericHelixController.java | 23 +++-
3 files changed, 27 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/3721c1fb/helix-core/src/main/java/org/apache/helix/HelixProperty.java
--
diff --git a/helix-core/src/main/java/org/apache/helix/HelixProperty.java
b/helix-core/src/main/java/org/apache/helix/HelixProperty.java
index 2f3e68d..02e0594 100644
--- a/helix-core/src/main/java/org/apache/helix/HelixProperty.java
+++ b/helix-core/src/main/java/org/apache/helix/HelixProperty.java
@@ -124,6 +124,12 @@ public class HelixProperty {
result = 31 * result + (int) (_modifiedTime ^ (_modifiedTime >>> 32));
return result;
}
+
+@Override
+public String toString() {
+ return "Stat {" + "_version=" + _version + ", _creationTime=" +
_creationTime
+ + ", _modifiedTime=" + _modifiedTime + '}';
+}
}
private Stat _stat;
@@ -180,7 +186,7 @@ public class HelixProperty {
@Override
public String toString() {
-return _record.toString();
+return "ZnRecord=" + _record.toString() + ", Stat=" + _stat.toString() ;
}
/**
http://git-wip-us.apache.org/repos/asf/helix/blob/3721c1fb/helix-core/src/main/java/org/apache/helix/common/caches/AbstractDataCache.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/common/caches/AbstractDataCache.java
b/helix-core/src/main/java/org/apache/helix/common/caches/AbstractDataCache.java
index 4bee84d..a4808b3 100644
---
a/helix-core/src/main/java/org/apache/helix/common/caches/AbstractDataCache.java
+++
b/helix-core/src/main/java/org/apache/helix/common/caches/AbstractDataCache.java
@@ -77,6 +77,8 @@ public abstract class AbstractDataCache {
}
}
+
+
List reloadedProperty = accessor.getProperty(reloadKeys, true);
Iterator csKeyIter = reloadKeys.iterator();
for (T property : reloadedProperty) {
@@ -88,6 +90,11 @@ public abstract class AbstractDataCache {
}
}
+LOG.info(reloadKeys.size() + " properties refreshed from zk.");
+if (LOG.isDebugEnabled()) {
+ LOG.debug("refreshed keys: " + reloadKeys);
+}
+
return refreshedPropertyMap;
}
http://git-wip-us.apache.org/repos/asf/helix/blob/3721c1fb/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
b/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
index eb75286..bd049f8 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
@@ -223,14 +223,13 @@ public class GenericHelixController implements
IdealStateChangeListener,
eventType, _clusterName));
}
- // TODO who should stop this timer
/**
* Starts the rebalancing timer with the specified period. Start the timer
if necessary; If the
* period is smaller than the current period, cancel the current timer and
use the new period.
*/
- void startRebalancingTimer(long period, HelixManager manager) {
+ void startPeriodRebalance(long period, HelixManager manager) {
if (period != _timerPeriod) {
- logger.info("Controller starting timer at period " + period);
+ logger.info("Controller starting periodical rebalance timer at period "
+ period);
if (_periodicalRebalanceTimer != null) {
_periodicalRebalanceTimer.cancel();
}
@@ -240,19 +239,21 @@ public class GenericHelixController implements
IdealStateChangeListener,
.scheduleAtFixedRate(new RebalanceTask(manager,
ClusterEventType.PeriodicalRebalance),
_timerPeriod, _timerPeriod);
} else {
- logger.info("Controller already has timer at period " + _tim
[8/8] helix git commit: Fix TestRoutingTableProviderFromTargetEV by using poll&wait check.
Fix TestRoutingTableProviderFromTargetEV by using poll&wait check.
Add a 3 seconds poll&wait for stabling the test.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/44d7d2eb
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/44d7d2eb
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/44d7d2eb
Branch: refs/heads/master
Commit: 44d7d2eb8b3da78630ca0d327d88448f8a82672c
Parents: 1507f01
Author: Jiajun Wang
Authored: Fri Oct 12 11:05:24 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:39:05 2018 -0700
--
.../TestRoutingTableProviderFromTargetEV.java | 20 +++-
1 file changed, 15 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/44d7d2eb/helix-core/src/test/java/org/apache/helix/integration/spectator/TestRoutingTableProviderFromTargetEV.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/spectator/TestRoutingTableProviderFromTargetEV.java
b/helix-core/src/test/java/org/apache/helix/integration/spectator/TestRoutingTableProviderFromTargetEV.java
index da6e2f8..6a24986 100644
---
a/helix-core/src/test/java/org/apache/helix/integration/spectator/TestRoutingTableProviderFromTargetEV.java
+++
b/helix-core/src/test/java/org/apache/helix/integration/spectator/TestRoutingTableProviderFromTargetEV.java
@@ -3,12 +3,14 @@ package org.apache.helix.integration.spectator;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
+
import org.apache.helix.ConfigAccessor;
import org.apache.helix.HelixException;
import org.apache.helix.HelixManager;
import org.apache.helix.HelixManagerFactory;
import org.apache.helix.InstanceType;
import org.apache.helix.PropertyType;
+import org.apache.helix.TestHelper;
import org.apache.helix.common.ZkTestBase;
import org.apache.helix.integration.manager.ClusterControllerManager;
import org.apache.helix.integration.manager.MockParticipantManager;
@@ -111,7 +113,7 @@ public class TestRoutingTableProviderFromTargetEV extends
ZkTestBase {
}
@Test (dependsOnMethods = "testTargetExternalViewWithoutEnable")
- public void testExternalViewDiffFromTargetExternalView() throws
InterruptedException {
+ public void testExternalViewDiffFromTargetExternalView() throws Exception {
ClusterConfig clusterConfig =
_configAccessor.getClusterConfig(CLUSTER_NAME);
clusterConfig.enableTargetExternalView(true);
clusterConfig.setPersistBestPossibleAssignment(true);
@@ -120,7 +122,7 @@ public class TestRoutingTableProviderFromTargetEV extends
ZkTestBase {
RoutingTableProvider externalViewProvider =
new RoutingTableProvider(_manager, PropertyType.EXTERNALVIEW);
-RoutingTableProvider targetExternalViewProvider =
+final RoutingTableProvider targetExternalViewProvider =
new RoutingTableProvider(_manager, PropertyType.TARGETEXTERNALVIEW);
try {
@@ -129,9 +131,17 @@ public class TestRoutingTableProviderFromTargetEV extends
ZkTestBase {
Set externalViewMasters =
externalViewProvider.getInstancesForResource(WorkflowGenerator.DEFAULT_TGT_DB,
"MASTER");
Assert.assertEquals(externalViewMasters.size(), 0);
- Set targetExternalViewMasters =
targetExternalViewProvider
- .getInstancesForResource(WorkflowGenerator.DEFAULT_TGT_DB, "MASTER");
- Assert.assertEquals(targetExternalViewMasters.size(), NUM_NODES);
+
+ final Set targetExternalViewMasters = new HashSet<>();
+ Assert.assertTrue(TestHelper.verify(new TestHelper.Verifier() {
+@Override
+public boolean verify() {
+ targetExternalViewMasters.clear();
+ targetExternalViewMasters.addAll(targetExternalViewProvider
+ .getInstancesForResource(WorkflowGenerator.DEFAULT_TGT_DB,
"MASTER"));
+ return targetExternalViewMasters.size() == NUM_NODES;
+}
+ }, 3000));
// TargetExternalView MASTERS mapping should exactly match IdealState
MASTERS mapping
Map> stateMap =
_gSetupTool.getClusterManagementTool()
[7/8] helix git commit: Improve helix message timeout task
Improve helix message timeout task
>From logs and code, it could be a very rare race condition that the message
>actually has been processed and completed but message has not been removed.
>Once it is completed, it should cancel the timeout task running with separated
>thread. But just before it tried to cancel the task, the message has been
>timed out and message handling thread has been interrupted by time out task
>thread, which shown in the log.
So the message handling thread did not catch the interrupted exception at that
moment and failed to remove message from ZK with READ state. After I manually
removed the message, we got an error log that showing the partition is already
LEADER now. That proves the assumption that the message has been successfully
process.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/1507f016
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/1507f016
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/1507f016
Branch: refs/heads/master
Commit: 1507f0161df24f4bec3ba3632b2d23a7a9bed5d4
Parents: c783ae7
Author: Junkai Xue
Authored: Wed Oct 10 16:59:28 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 14:38:58 2018 -0700
--
.../helix/messaging/handling/HelixTask.java | 28 +---
1 file changed, 18 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/1507f016/helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTask.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTask.java
b/helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTask.java
index 2f3d805..fb55e76 100644
---
a/helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTask.java
+++
b/helix-core/src/main/java/org/apache/helix/messaging/handling/HelixTask.java
@@ -22,7 +22,6 @@ package org.apache.helix.messaging.handling;
import java.util.Date;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
-
import org.apache.helix.AccessOption;
import org.apache.helix.HelixDataAccessor;
import org.apache.helix.HelixManager;
@@ -96,6 +95,9 @@ public class HelixTask implements MessageTask {
handlerStart = System.currentTimeMillis();
taskResult = _handler.handleMessage();
handlerEnd = System.currentTimeMillis();
+
+ // cancel timeout task
+ _executor.cancelTimeoutTask(this);
} catch (InterruptedException e) {
taskResult = new HelixTaskResult();
taskResult.setException(e);
@@ -116,9 +118,6 @@ public class HelixTask implements MessageTask {
_statusUpdateUtil.logError(_message, HelixTask.class, e, errorMessage,
_manager);
}
-// cancel timeout task
-_executor.cancelTimeoutTask(this);
-
Exception exception = null;
try {
if (taskResult.isSuccess()) {
@@ -182,13 +181,9 @@ public class HelixTask implements MessageTask {
}
}
- if (_message.getAttribute(Attributes.PARENT_MSG_ID) == null) {
-removeMessageFromZk(accessor, _message);
-reportMessageStat(_manager, _message, taskResult);
-sendReply(getSrcClusterDataAccessor(_message), _message, taskResult);
-_executor.finishTask(this);
- }
+ finalCleanup(taskResult);
} catch (Exception e) {
+ finalCleanup(taskResult);
exception = e;
type = ErrorType.FRAMEWORK;
code = ErrorCode.ERROR;
@@ -377,4 +372,17 @@ public class HelixTask implements MessageTask {
}
_isStarted = true;
}
+
+ private void finalCleanup(HelixTaskResult taskResult) {
+try {
+ if (_message.getAttribute(Attributes.PARENT_MSG_ID) == null) {
+removeMessageFromZk(_manager.getHelixDataAccessor(), _message);
+reportMessageStat(_manager, _message, taskResult);
+sendReply(getSrcClusterDataAccessor(_message), _message, taskResult);
+_executor.finishTask(this);
+ }
+} catch (Exception e) {
+ logger.error(String.format("Error to final clean up for message : %s",
_message.getId()));
+}
+ }
}
[1/3] helix git commit: UI: Front page minor adjustment; prepare yarn migration
Repository: helix Updated Branches: refs/heads/master 44d7d2eb8 -> b64a72c48 http://git-wip-us.apache.org/repos/asf/helix/blob/b64a72c4/helix-front/yarn.lock -- diff --git a/helix-front/yarn.lock b/helix-front/yarn.lock index a12f972..cf418cb 100644 --- a/helix-front/yarn.lock +++ b/helix-front/yarn.lock @@ -2,681 +2,962 @@ # yarn lockfile v1 -"@angular/animations@^4.3.2": - version "4.3.3" - resolved "https://registry.yarnpkg.com/@angular/animations/-/animations-4.3.3.tgz#b71ddd453673929f550b171cca99952b3aaa831c"; +"@angular-devkit/build-optimizer@0.3.2": + version "0.3.2" + resolved "https://registry.yarnpkg.com/@angular-devkit/build-optimizer/-/build-optimizer-0.3.2.tgz#77d4d1e4fcd09575f0200e2d6d21c1a3f995096f"; + integrity sha512-U0BCZtThq5rUfY08shHXpxe8ZhSsiYB/cJjUvAWRTs/ORrs8pbngS6xwseQws8d/vHoVrtqGD9GU9h8AmFRERQ== + dependencies: +loader-utils "^1.1.0" +source-map "^0.5.6" +typescript "~2.6.2" +webpack-sources "^1.0.1" + +"@angular-devkit/core@0.3.2": + version "0.3.2" + resolved "https://registry.yarnpkg.com/@angular-devkit/core/-/core-0.3.2.tgz#b6fd6aa2e4f4ac3c39478198ba91a53d4c0f88d5"; + integrity sha512-zABk/iP7YX5SVbmK4e+IX7j2d0D37MQJQiKgWdV3JzfvVJhNJzddiirtT980pIafoq+KyvTgVwXtc+vnux0oeQ== + dependencies: +ajv "~5.5.1" +chokidar "^1.7.0" +rxjs "^5.5.6" +source-map "^0.5.6" + +"@angular-devkit/schematics@0.3.2": + version "0.3.2" + resolved "https://registry.yarnpkg.com/@angular-devkit/schematics/-/schematics-0.3.2.tgz#09a2268fad61f4f956f42db87913b8748c512a78"; + integrity sha512-B6zZoqvHaTJy+vVdA6EtlxnCdGMa5elCa4j9lQLC3JI8DLvMXUWkCIPVbPzJ/GSRR9nsKWpvYMYaJyfBDUqfhw== + dependencies: +"@ngtools/json-schema" "^1.1.0" +rxjs "^5.5.6" + +"@angular/animations@^5.1.1": + version "5.2.11" + resolved "https://registry.yarnpkg.com/@angular/animations/-/animations-5.2.11.tgz#2bd3fe9e72916ca28de9bfaaddf0cb936565a0b8"; + integrity sha512-J7wKHkFn3wV28/Y1Qm4yjGXVCwXzj1JR5DRjGDTFnxTRacUFx7Nj0ApGhN0b2+V0NOvgxQOvEW415Y22kGoblw== dependencies: tslib "^1.7.1" -"@angular/cli@^1.2.0": - version "1.2.7" - resolved "https://registry.yarnpkg.com/@angular/cli/-/cli-1.2.7.tgz#dd20e8b0da24af5359077c05a6944823cb764132"; +"@angular/cdk@^5.0.1": + version "5.2.5" + resolved "https://registry.yarnpkg.com/@angular/cdk/-/cdk-5.2.5.tgz#cae2b12e1990a692dd267a73fdb1d49db37f9604"; + integrity sha512-GN8m1d+VcCE9+Bgwv06Y8YJKyZ0i9ZIq2ZPBcJYt+KVgnVVRg4JkyUNxud07LNsvzOX22DquHqmIZiC4hAG7Ag== dependencies: -"@ngtools/json-schema" "1.1.0" -"@ngtools/webpack" "1.5.5" -autoprefixer "^6.5.3" -chalk "^2.0.1" -circular-dependency-plugin "^3.0.0" +tslib "^1.7.1" + +"@angular/cli@^1.6.1": + version "1.7.4" + resolved "https://registry.yarnpkg.com/@angular/cli/-/cli-1.7.4.tgz#b6c31b5fc6f8ea07e55b1b01a26422f5358a4ea6"; + integrity sha512-URdb1QtnQf+Ievy93wjq7gE81s25BkWUwJFPey+YkphBA3G1lbCAQPiEh2pntBwaIKavgEuCw+Sf2YZdgTVhDA== + dependencies: +"@angular-devkit/build-optimizer" "0.3.2" +"@angular-devkit/core" "0.3.2" +"@angular-devkit/schematics" "0.3.2" +"@ngtools/json-schema" "1.2.0" +"@ngtools/webpack" "1.10.2" +"@schematics/angular" "0.3.2" +"@schematics/package-update" "0.3.2" +ajv "^6.1.1" +autoprefixer "^7.2.3" +cache-loader "^1.2.0" +chalk "~2.2.0" +circular-dependency-plugin "^4.2.1" +clean-css "^4.1.11" common-tags "^1.3.1" +copy-webpack-plugin "~4.4.1" core-object "^3.1.0" -css-loader "^0.28.1" -cssnano "^3.10.0" denodeify "^1.2.1" -diff "^3.1.0" -ember-cli-normalize-entity-name "^1.0.0" ember-cli-string-utils "^1.0.0" -exports-loader "^0.6.3" -extract-text-webpack-plugin "^2.1.0" -file-loader "^0.10.0" +extract-text-webpack-plugin "^3.0.2" +file-loader "^1.1.5" fs-extra "^4.0.0" -get-caller-file "^1.0.0" glob "^7.0.3" -heimdalljs "^0.2.4" -heimdalljs-logger "^0.1.9" -html-webpack-plugin "^2.19.0" -inflection "^1.7.0" -inquirer "^3.0.0" -isbinaryfile "^3.0.0" -istanbul-instrumenter-loader "^2.0.0" +html-webpack-plugin "^2.29.0" +istanbul-instrumenter-loader "^3.0.0" karma-source-map-support "^1.2.0" less "^2.7.2" -less-loader "^4.0.2" -license-webpack-plugin "^0.4.2" +less-loader "^4.0.5" +license-webpack-plugin "^1.0.0" +loader-utils "1.1.0" lodash "^4.11.1" memory-fs "^0.4.1" -minimatch "^3.0.3" +minimatch "^3.0.4" node-modules-path "^1.0.0" nopt "^4.0.1" opn "~5.1.0" portfinder "~1.0.12" -postcss-loader "^1.3.3" -postcss-url "^5.1.2" +postcss "^6.0.16" +postcss-import "^11.0.0" +postcss-loader "^2.0.10" +postcss-url "^7.1.2" raw-loader "^0.5.1" resolve "^1.1.7" -rsvp "^3.0.17" -rxjs "^5.0.1" -sass-loader "^6.0.3" -script-loader "^0.7.0" +rxjs "^5.5.6" +sass-loader
[3/3] helix git commit: UI: Front page minor adjustment; prepare yarn migration
UI: Front page minor adjustment; prepare yarn migration
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/b64a72c4
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/b64a72c4
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/b64a72c4
Branch: refs/heads/master
Commit: b64a72c48ee393619da518fcb4ab96d2106ad22d
Parents: 44d7d2e
Author: Vivo Xu
Authored: Sun Oct 21 17:52:21 2018 -0700
Committer: Vivo Xu
Committed: Thu Nov 1 16:04:12 2018 -0700
--
.../helix-list/helix-list.component.html|2 +-
.../helix-list/helix-list.component.scss|4 +
helix-front/package-lock.json | 4000 +--
helix-front/package.json|2 +-
helix-front/yarn.lock | 6616 --
5 files changed, 6292 insertions(+), 4332 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/b64a72c4/helix-front/client/app/chooser/helix-list/helix-list.component.html
--
diff --git
a/helix-front/client/app/chooser/helix-list/helix-list.component.html
b/helix-front/client/app/chooser/helix-list/helix-list.component.html
index e8df780..5e494a4 100644
--- a/helix-front/client/app/chooser/helix-list/helix-list.component.html
+++ b/helix-front/client/app/chooser/helix-list/helix-list.component.html
@@ -1,6 +1,6 @@
-{{ group }}
+{{ group | titlecase }}
http://git-wip-us.apache.org/repos/asf/helix/blob/b64a72c4/helix-front/client/app/chooser/helix-list/helix-list.component.scss
--
diff --git
a/helix-front/client/app/chooser/helix-list/helix-list.component.scss
b/helix-front/client/app/chooser/helix-list/helix-list.component.scss
index 97e941b..36bcba6 100644
--- a/helix-front/client/app/chooser/helix-list/helix-list.component.scss
+++ b/helix-front/client/app/chooser/helix-list/helix-list.component.scss
@@ -6,3 +6,7 @@
text-align: left;
}
}
+
+.mat-card-title {
+ font-size: 20px;
+}
[2/3] helix git commit: UI: Front page minor adjustment; prepare yarn migration
http://git-wip-us.apache.org/repos/asf/helix/blob/b64a72c4/helix-front/package-lock.json
--
diff --git a/helix-front/package-lock.json b/helix-front/package-lock.json
index 6398e0c..015140f 100644
--- a/helix-front/package-lock.json
+++ b/helix-front/package-lock.json
@@ -1,6 +1,6 @@
{
"name": "helix-front",
- "version": "1.2.0",
+ "version": "1.2.1",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
@@ -10,10 +10,10 @@
"integrity":
"sha512-EFFF7hBbVoTOzYfXuSlGhcDr8neafmwuBAIkzAekEjzik7OaTLq7LPG7As+ebed9ll+3DAGypnrpdIE1Tp/H/A==",
"dev": true,
"requires": {
-"loader-utils": "^1.1.0",
-"source-map": "^0.5.6",
-"typescript": "~2.6.1",
-"webpack-sources": "^1.0.1"
+"loader-utils": "1.1.0",
+"source-map": "0.5.7",
+"typescript": "2.6.2",
+"webpack-sources": "1.1.0"
},
"dependencies": {
"source-map": {
@@ -36,7 +36,7 @@
"integrity":
"sha512-zxrNtTiv60liye/GGeRMnnGgLgAWoqlMTfPLMW0D1qJ4bbrPHtme010mpxS3QL4edcDtQseyXSFCnEkuo2MrRw==",
"dev": true,
"requires": {
-"source-map": "^0.5.6"
+"source-map": "0.5.7"
},
"dependencies": {
"source-map": {
@@ -54,10 +54,10 @@
"dev": true,
"requires": {
"@angular-devkit/core": "0.0.22",
-"@ngtools/json-schema": "^1.1.0",
+"@ngtools/json-schema": "1.1.0",
"@schematics/schematics": "0.0.11",
-"minimist": "^1.2.0",
-"rxjs": "^5.5.2"
+"minimist": "1.2.0",
+"rxjs": "5.5.5"
}
},
"@angular/animations": {
@@ -65,7 +65,7 @@
"resolved":
"https://registry.npmjs.org/@angular/animations/-/animations-5.1.1.tgz";,
"integrity":
"sha512-PHLBWDnAzr5b5l52pk5ZYmv/6m0YUe2ICwu5dmbS0d8Kf5dXadMphAWCDbljMF+djGyZeFq2/dQ/t7ygYl3YuA==",
"requires": {
-"tslib": "^1.7.1"
+"tslib": "1.8.1"
}
},
"@angular/cdk": {
@@ -73,7 +73,7 @@
"resolved": "https://registry.npmjs.org/@angular/cdk/-/cdk-5.0.1.tgz";,
"integrity":
"sha512-uK4Vyaf06J8KqePzq35BxMHRGolt35EnbZf9wjCs7eYaghbQ7Pk2xUGoynu5Lj1wAOn5N1/C1nT2/aAH/EE2rw==",
"requires": {
-"tslib": "^1.7.1"
+"tslib": "1.8.1"
}
},
"@angular/cli": {
@@ -82,64 +82,64 @@
"integrity":
"sha512-cdXfUR1by7nCgg9H9HijV8aI36AmceMubBZ/k+UgICMd8DlkPCVBIWcUXLdoH/l6nuXb3gCucfJoThznlakNMw==",
"dev": true,
"requires": {
-"@angular-devkit/build-optimizer": "~0.0.36",
-"@angular-devkit/schematics": "~0.0.42",
+"@angular-devkit/build-optimizer": "0.0.36",
+"@angular-devkit/schematics": "0.0.42",
"@ngtools/json-schema": "1.1.0",
"@ngtools/webpack": "1.9.1",
-"@schematics/angular": "~0.1.11",
-"autoprefixer": "^6.5.3",
-"chalk": "~2.2.0",
-"circular-dependency-plugin": "^4.2.1",
-"common-tags": "^1.3.1",
-"copy-webpack-plugin": "^4.1.1",
-"core-object": "^3.1.0",
-"css-loader": "^0.28.1",
-"cssnano": "^3.10.0",
-"denodeify": "^1.2.1",
-"ember-cli-string-utils": "^1.0.0",
-"exports-loader": "^0.6.3",
-"extract-text-webpack-plugin": "^3.0.2",
-"file-loader": "^1.1.5",
-"fs-extra": "^4.0.0",
-"glob": "^7.0.3",
-"html-webpack-plugin": "^2.29.0",
-"istanbul-instrumenter-loader": "^2.0.0",
-"karma-source-map-support": "^1.2.0",
-"less": "^2.7.2",
-"less-loader": "^4.0.5",
-"license-webpack-plugin": "^1.0.0",
+"@schematics/angular": "0.1.11",
+"autoprefixer": "6.7.7",
+"chalk": "2.2.2",
+"circular-dependency-plugin": "4.3.0",
+"common-tags": "1.5.1",
+"copy-webpack-plugin": "4.3.0",
+"core-object": "3.1.5",
+"css-loader": "0.28.7",
+"cssnano": "3.10.0",
+"denodeify": "1.2.1",
+"ember-cli-string-utils": "1.1.0",
+"exports-loader": "0.6.4",
+"extract-text-webpack-plugin": "3.0.2",
+"file-loader": "1.1.5",
+"fs-extra": "4.0.3",
+"glob": "7.1.2",
+"html-webpack-plugin": "2.30.1",
+"istanbul-instrumenter-loader": "2.0.0",
+"karma-source-map-support": "1.2.0",
+"less": "2.7.3",
+"less-loader": "4.0.5",
+"license-webpack-plugin": "1.1.1",
"loader-utils": "1.1.0",
-"lodash": "^4.11.1",
-"memory-fs": "^0.4.1",
-"minimatch": "^3.0.4",
-"node-modules-path": "^1.0.0",
-"node-sass": "^4.3.0",
-"nopt": "^4.0.1",
-"opn": "~5.1.0",
-"portfinder": "~1.0.12",
-"postcss-custom-properties": "^6.1.0",
-"postcss-loader": "^2.0.8",
-"postcss-url": "^7.1.2",
-"raw-loader": "^0.5.1",
-"resolve": "^1.1.7",
-"rxjs": "^5.5.2",
-"
[jira] [Created] (HELIX-782) TASK: Make TaskDriver use ZKClient's create when creating workflows
Hunter L created HELIX-782: -- Summary: TASK: Make TaskDriver use ZKClient's create when creating workflows Key: HELIX-782 URL: https://issues.apache.org/jira/browse/HELIX-782 Project: Apache Helix Issue Type: Improvement Reporter: Hunter L Assignee: Hunter L TaskDriver should use create() but currently is using set(), which just overwrites ZNodes that are in ZK. This is undesirable and we need to fix it, especially in the wake of ZNode restructuring. AC: 1. Make TaskDriver use create() instead of set() 2. Add an integration test: TestWorkflowCreation:testWorkflowCreationNoDuplicates() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HELIX-783) TASK: Fix JobQueue's job state-related bug
Hunter L created HELIX-783: -- Summary: TASK: Fix JobQueue's job state-related bug Key: HELIX-783 URL: https://issues.apache.org/jira/browse/HELIX-783 Project: Apache Helix Issue Type: Improvement Reporter: Hunter L Assignee: Hunter L The bug was observed in TestTaskRebalancerStopResume:stopAndResumeNamedQueue(), which was being unstable. It was observed that for JobQueues with multiple jobs, the second job would get marked as IN_PROGRESS even though the first job hadn't completed/failed, especially when the queue was being stopped and resumed. This was due to a bug in getIncompleteJobCount() because it was not counting jobs in STOPPING state. This was fixed and another check was added right before JobDispatcher marks a job as STOPPED so that it would not mark it STOPPED if the job state is NOT_STARTED. Changelist: 1. Fix getIncompleteJobCount() 2. Add a check so that we don't mark NOT_STARTED jobs as STOPPED -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HELIX-784) TASK: Fix a bug in getExpiredJobs
Hunter L created HELIX-784: -- Summary: TASK: Fix a bug in getExpiredJobs Key: HELIX-784 URL: https://issues.apache.org/jira/browse/HELIX-784 Project: Apache Helix Issue Type: Improvement Reporter: Hunter L Assignee: Hunter L getExpiredJobs(), when the job config is null, would just continue instead of adding it to expiredJobs so that the job cleanup/purge would be re-tried. This could possibly cause purge failures to leave a lot of jobs un-purged with just the job config missing in ZK. This RB fixes this. Changelist: 1. Add the job name to expiredJobs if the job config does not exist in ZK 2. Add a more detailed description in the error log 3. Add an integration test for two task-related stages: TaskPersistDataStage and TaskGarbageCollectionStage in TestTaskStage.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[2/3] helix git commit: TASK: Fix JobQueue's job state-related bug
TASK: Fix JobQueue's job state-related bug
The bug was observed in TestTaskRebalancerStopResume:stopAndResumeNamedQueue(),
which was being unstable.
It was observed that for JobQueues with multiple jobs, the second job would get
marked as IN_PROGRESS even though the first job hadn't completed/failed,
especially when the queue was being stopped and resumed. This was due to a bug
in getIncompleteJobCount() because it was not counting jobs in STOPPING state.
This was fixed and another check was added right before JobDispatcher marks a
job as STOPPED so that it would not mark it STOPPED if the job state is
NOT_STARTED.
Changelist:
1. Fix getIncompleteJobCount()
2. Add a check so that we don't mark NOT_STARTED jobs as STOPPED
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/3d9c0306
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/3d9c0306
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/3d9c0306
Branch: refs/heads/master
Commit: 3d9c03064a5c26a9ed9ad674567674f2d9eca160
Parents: 3844ad6
Author: narendly
Authored: Thu Nov 1 16:55:59 2018 -0700
Committer: narendly
Committed: Thu Nov 1 16:55:59 2018 -0700
--
helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java | 2 +-
helix-core/src/main/java/org/apache/helix/task/TaskUtil.java | 3 ++-
.../org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java | 3 ++-
3 files changed, 5 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/3d9c0306/helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java
--
diff --git a/helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java
b/helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java
index 06f6ce4..05d8e5c 100644
--- a/helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java
+++ b/helix-core/src/main/java/org/apache/helix/task/JobDispatcher.java
@@ -133,7 +133,7 @@ public class JobDispatcher extends AbstractTaskDispatcher {
// TIMING_OUT/FAILING/ABORTING job can't be stopped, because all tasks
are being aborted
// Update running status in workflow context
if (jobTgtState == TargetState.STOP) {
-if (TaskUtil.checkJobStopped(jobCtx)) {
+if (jobState != TaskState.NOT_STARTED &&
TaskUtil.checkJobStopped(jobCtx)) {
workflowCtx.setJobState(jobName, TaskState.STOPPED);
} else {
workflowCtx.setJobState(jobName, TaskState.STOPPING);
http://git-wip-us.apache.org/repos/asf/helix/blob/3d9c0306/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
--
diff --git a/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
b/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
index 958805b..b587408 100644
--- a/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
+++ b/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
@@ -943,7 +943,8 @@ public class TaskUtil {
int inCompleteCount = 0;
for (String jobName : workflowCfg.getJobDag().getAllNodes()) {
TaskState jobState = workflowCtx.getJobState(jobName);
- if (jobState == TaskState.IN_PROGRESS || jobState == TaskState.STOPPED) {
+ if (jobState == TaskState.IN_PROGRESS || jobState == TaskState.STOPPED
+ || jobState == TaskState.STOPPING) {
++inCompleteCount;
}
}
http://git-wip-us.apache.org/repos/asf/helix/blob/3d9c0306/helix-core/src/test/java/org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java
b/helix-core/src/test/java/org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java
index 73fe674..25c486a 100644
---
a/helix-core/src/test/java/org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java
+++
b/helix-core/src/test/java/org/apache/helix/task/TestGetLastScheduledTaskExecInfo.java
@@ -53,7 +53,7 @@ public class TestGetLastScheduledTaskExecInfo extends
TaskTestBase {
Assert.assertEquals(startTimesWithStuckTasks.get(3), lastScheduledTaskTs);
Assert.assertEquals(execInfo.getJobName(), "TestWorkflow_2_job_0");
-// Workflow 2 will stuck, so its partition state will be RUNNING
+// Workflow 2 will be stuck, so its partition state will be RUNNING
Assert.assertEquals(execInfo.getTaskPartitionState(),
TaskPartitionState.RUNNING);
Assert.assertEquals(execInfo.getStartTimeStamp(), lastScheduledTaskTs);
@@ -61,6 +61,7 @@ public class TestGetLastScheduledTaskExecInfo extends
TaskTestBase {
// API call needs to return the most recent timestamp (value at last index)
[3/3] helix git commit: [HELIX-784] TASK: Fix a bug in getExpiredJobs
[HELIX-784] TASK: Fix a bug in getExpiredJobs
getExpiredJobs(), when the job config is null, would just continue instead of
adding it to expiredJobs so that the job cleanup/purge would be re-tried. This
could possibly cause purge failures to leave a lot of jobs un-purged with just
the job config missing in ZK. This RB fixes this.
Changelist:
1. Add the job name to expiredJobs if the job config does not exist in ZK
2. Add a more detailed description in the error log
3. Add an integration test for two task-related stages: TaskPersistDataStage
and TaskGarbageCollectionStage in TestTaskStage.java
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/befb1036
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/befb1036
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/befb1036
Branch: refs/heads/master
Commit: befb1036f8d8be2729a800d3dde88fc1362a6489
Parents: 3d9c030
Author: narendly
Authored: Thu Nov 1 16:57:33 2018 -0700
Committer: narendly
Committed: Thu Nov 1 16:57:33 2018 -0700
--
.../java/org/apache/helix/task/TaskUtil.java| 6 +-
.../stages/TestTaskPersistDataStage.java| 91 --
.../helix/controller/stages/TestTaskStage.java | 165 +++
3 files changed, 170 insertions(+), 92 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/befb1036/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
--
diff --git a/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
b/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
index b587408..8392d89 100644
--- a/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
+++ b/helix-core/src/main/java/org/apache/helix/task/TaskUtil.java
@@ -719,7 +719,11 @@ public class TaskUtil {
JobConfig jobConfig = TaskUtil.getJobConfig(dataAccessor, job);
JobContext jobContext = TaskUtil.getJobContext(propertyStore, job);
if (jobConfig == null) {
- LOG.error(String.format("Job %s exists in JobDAG but JobConfig is
missing!", job));
+ LOG.error(String.format(
+ "Job %s exists in JobDAG but JobConfig is missing! Job might
have been deleted manually from the JobQueue: %s, or left in the DAG due to a
failed clean-up attempt from last purge.",
+ job, workflowConfig.getWorkflowId()));
+ // Add the job name to expiredJobs so that purge operation will be
tried again on this job
+ expiredJobs.add(job);
continue;
}
long expiry = jobConfig.getExpiry();
http://git-wip-us.apache.org/repos/asf/helix/blob/befb1036/helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskPersistDataStage.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskPersistDataStage.java
b/helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskPersistDataStage.java
deleted file mode 100644
index 8fcedfa..000
---
a/helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskPersistDataStage.java
+++ /dev/null
@@ -1,91 +0,0 @@
-package org.apache.helix.controller.stages;
-
-import org.apache.helix.ConfigAccessor;
-import org.apache.helix.HelixManager;
-import org.apache.helix.InstanceType;
-import org.apache.helix.TestHelper;
-import org.apache.helix.ZNRecord;
-import org.apache.helix.common.ZkTestBase;
-import org.apache.helix.common.caches.TaskDataCache;
-import org.apache.helix.controller.stages.task.TaskPersistDataStage;
-import org.apache.helix.participant.MockZKHelixManager;
-import org.apache.helix.task.JobContext;
-import org.apache.helix.task.TaskDriver;
-import org.apache.helix.task.TaskPartitionState;
-import org.apache.helix.task.TaskState;
-import org.apache.helix.task.Workflow;
-import org.apache.helix.task.WorkflowContext;
-import org.testng.Assert;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.Test;
-
-public class TestTaskPersistDataStage extends ZkTestBase {
- private String CLUSTER_NAME = "TestCluster_" + TestHelper.getTestClassName();
- private HelixManager _helixManager;
- private TaskDriver _driver;
-
- @BeforeClass
- public void beforeClass() {
-_helixManager = new MockZKHelixManager(CLUSTER_NAME, "MockInstance",
InstanceType.ADMINISTRATOR,
-_gZkClient);
-_driver = new TaskDriver(_gZkClient, CLUSTER_NAME);
- }
-
- @Test
- public void testTaskContextUpdate() {
-ClusterEvent event = new ClusterEvent(CLUSTER_NAME,
ClusterEventType.CurrentStateChange);
-event.addAttribute(AttributeName.helixmanager.name(), _helixManager);
-TaskPersistDataStage persistDataStage = new TaskPersistDataStage();
-
-ClusterDataCache cache
[1/3] helix git commit: [HELIX-782] TASK: Make TaskDriver use ZKClient's create when creating workflows
Repository: helix
Updated Branches:
refs/heads/master b64a72c48 -> befb1036f
[HELIX-782] TASK: Make TaskDriver use ZKClient's create when creating workflows
TaskDriver should use create() but currently is using set(), which just
overwrites ZNodes that are in ZK. This is undesirable and we need to fix it,
especially in the wake of ZNode restructuring.
AC:
1. Make TaskDriver use create() instead of set()
2. Add an integration test:
TestWorkflowCreation:testWorkflowCreationNoDuplicates()
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/3844ad60
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/3844ad60
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/3844ad60
Branch: refs/heads/master
Commit: 3844ad60034b029f3bbd916f629a7969117c1b26
Parents: b64a72c
Author: narendly
Authored: Thu Nov 1 16:54:48 2018 -0700
Committer: narendly
Committed: Thu Nov 1 16:54:48 2018 -0700
--
.../java/org/apache/helix/task/TaskDriver.java | 27 +-
.../java/org/apache/helix/task/TaskUtil.java| 32 ++--
.../apache/helix/task/TestWorkflowCreation.java | 55
3 files changed, 99 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/3844ad60/helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
--
diff --git a/helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
b/helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
index 27670e9..a522a86 100644
--- a/helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
+++ b/helix-core/src/main/java/org/apache/helix/task/TaskDriver.java
@@ -129,7 +129,6 @@ public class TaskDriver {
* @param flow
*/
public void start(Workflow flow) {
-// TODO: check that namespace for workflow is available
LOG.info("Starting workflow " + flow.getName());
flow.validate();
@@ -154,8 +153,17 @@ public class TaskDriver {
newWorkflowConfig.setJobTypes(jobTypes);
// add workflow config.
-if (!TaskUtil.setWorkflowConfig(_accessor, flow.getName(),
newWorkflowConfig)) {
- LOG.error("Failed to add workflow configuration for workflow " +
flow.getName());
+if (!TaskUtil.createWorkflowConfig(_accessor, flow.getName(),
newWorkflowConfig)) {
+ // workflow config creation failed; try to delete job configs back
+ Set failedJobRemoval = new HashSet<>();
+ for (String job : flow.getJobConfigs().keySet()) {
+if (!TaskUtil.removeJobConfig(_accessor, job)) {
+ failedJobRemoval.add(job);
+}
+ }
+ throw new HelixException(String.format(
+ "Failed to add workflow configuration for workflow %s. It's possible
that a workflow of the same name already exists or there was a connection
issue. JobConfig deletion attempted but failed for the following jobs: %s",
+ flow.getName(), failedJobRemoval));
}
// Finally add workflow resource.
@@ -171,19 +179,14 @@ public class TaskDriver {
* the new configuration will be applied to the next scheduled runs of the
workflow.
* For non-recurrent workflow, the new configuration may (or may not) be
applied
* on the current running jobs, but it will be applied on the following
unscheduled jobs.
- *
* Example:
- *
* _driver = new TaskDriver ...
* WorkflowConfig currentWorkflowConfig = _driver.getWorkflowCfg(_manager,
workflow);
* WorkflowConfig.Builder configBuilder = new
WorkflowConfig.Builder(currentWorkflowConfig);
-
* // make needed changes to the config here
* configBuilder.setXXX();
- *
* // update workflow configuration
* _driver.updateWorkflow(workflow, configBuilder.build());
- *
* @param workflow
* @param newWorkflowConfig
*/
@@ -575,21 +578,21 @@ public class TaskDriver {
}
/**
- * Add new job config to cluster
+ * Add new job config to cluster by way of create
*/
private void addJobConfig(String job, JobConfig jobConfig) {
LOG.info("Add job configuration " + job);
// Set the job configuration
JobConfig newJobCfg = new JobConfig(job, jobConfig);
-if (!TaskUtil.setJobConfig(_accessor, job, newJobCfg)) {
- throw new HelixException("Failed to add job configuration for job " +
job);
+if (!TaskUtil.createJobConfig(_accessor, job, newJobCfg)) {
+ throw new HelixException("Failed to add job configuration for job " + job
+ + ". It's possible that a job of the same name already exists or
there was a connection issue");
}
}
/**
* Public method to resume a workflow/queue.
- *
* @param workflow
*/
public void resume(String workflow) {
http://git-wip-us.apache.org/repos/asf/helix/blob/3844ad60/helix-core/src/main/java
[2/3] helix git commit: Fix unstable test disable partition
Fix unstable test disable partition
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/0f0d4177
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/0f0d4177
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/0f0d4177
Branch: refs/heads/master
Commit: 0f0d4177e830f9f1f3a5f26abeae0b366e9b1863
Parents: 1e1cc41
Author: Junkai Xue
Authored: Thu Oct 25 10:55:19 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 17:10:41 2018 -0700
--
.../org/apache/helix/integration/manager/TestZkHelixAdmin.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/0f0d4177/helix-core/src/test/java/org/apache/helix/integration/manager/TestZkHelixAdmin.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/manager/TestZkHelixAdmin.java
b/helix-core/src/test/java/org/apache/helix/integration/manager/TestZkHelixAdmin.java
index c2da24d..5141a8d 100644
---
a/helix-core/src/test/java/org/apache/helix/integration/manager/TestZkHelixAdmin.java
+++
b/helix-core/src/test/java/org/apache/helix/integration/manager/TestZkHelixAdmin.java
@@ -60,6 +60,8 @@ public class TestZkHelixAdmin extends TaskTestBase {
public void testEnableDisablePartitions() throws InterruptedException {
_admin.enablePartition(false, CLUSTER_NAME, (PARTICIPANT_PREFIX + "_" +
_startPort),
WorkflowGenerator.DEFAULT_TGT_DB, Arrays.asList(new String[] {
"TestDB_0", "TestDB_2" }));
+_admin.enablePartition(false, CLUSTER_NAME, (PARTICIPANT_PREFIX + "_" +
(_startPort + 1)),
+WorkflowGenerator.DEFAULT_TGT_DB, Arrays.asList(new String[] {
"TestDB_0", "TestDB_2" }));
IdealState idealState =
_admin.getResourceIdealState(CLUSTER_NAME,
WorkflowGenerator.DEFAULT_TGT_DB);
[1/3] helix git commit: Fix redundant DROPPED message sent to participant
Repository: helix
Updated Branches:
refs/heads/master befb1036f -> 73c3f0ad8
Fix redundant DROPPED message sent to participant
It was caused by combination of two Helix logic:
1. Helix caches best possible mapping and wont recompute it unless there are
changes to IdealState, LiveInstance, ResourceConfig or InstanceConfig .
2. In message generation, if current state does not exist, Helix will think it
is in INITIAL (OFFLINE) state
In this case, we have two fixes for that:
1. If we see current state is null and target state is DROPPED, Helix will not
send OFFLINE -> DROPPED message anymore.
2. if we see recurrent OFFLINE -> DROPPED message, Helix will clean up the
cached best possible mapping for this resource and let it recompute.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/1e1cc41e
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/1e1cc41e
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/1e1cc41e
Branch: refs/heads/master
Commit: 1e1cc41ea1c2d911ee8d010495121bd73e5fb4d2
Parents: befb103
Author: Junkai Xue
Authored: Wed Oct 17 16:26:38 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 17:10:35 2018 -0700
--
.../controller/stages/ClusterDataCache.java | 8 +++
.../stages/MessageGenerationPhase.java | 8 +++
.../controller/TestRedundantDroppedMessage.java | 72
.../paticipant/TestStateTransitionTimeout.java | 2 +-
.../TestStateTransitionTimeoutWithResource.java | 2 +-
5 files changed, 90 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/1e1cc41e/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
b/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
index 6de6d51..08e98cc 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
@@ -871,6 +871,14 @@ public class ClusterDataCache extends AbstractDataCache {
}
/**
+ * Invalid the cached resourceAssignment (ideal mapping) for a resource
+ * @param resource
+ */
+ public void invalidCachedIdealStateMapping(String resource) {
+_idealMappingCache.remove(resource);
+ }
+
+ /**
* Get cached idealmapping
* @return
*/
http://git-wip-us.apache.org/repos/asf/helix/blob/1e1cc41e/helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
b/helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
index b1013d1..c829082 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationPhase.java
@@ -28,6 +28,7 @@ import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixDefinedState;
import org.apache.helix.HelixException;
import org.apache.helix.HelixManager;
import org.apache.helix.SystemPropertyKeys;
@@ -150,6 +151,13 @@ public abstract class MessageGenerationPhase extends
AbstractBaseStage {
currentStateOutput.getCurrentState(resourceName, partition,
instanceName);
if (currentState == null) {
currentState = stateModelDef.getInitialState();
+ if (desiredState.equals(HelixDefinedState.DROPPED.name())) {
+LogUtil.logDebug(logger, _eventId, String
+.format("No current state for partition %s in resource %s,
skip the drop message",
+partition.getPartitionName(), resourceName));
+cache.invalidCachedIdealStateMapping(resourceName);
+continue;
+ }
}
Message pendingMessage =
http://git-wip-us.apache.org/repos/asf/helix/blob/1e1cc41e/helix-core/src/test/java/org/apache/helix/integration/controller/TestRedundantDroppedMessage.java
--
diff --git
a/helix-core/src/test/java/org/apache/helix/integration/controller/TestRedundantDroppedMessage.java
b/helix-core/src/test/java/org/apache/helix/integration/controller/TestRedundantDroppedMessage.java
new file mode 100644
index 000..438ad4f
--- /dev/null
+++
b/helix-core/src/test/java/org/apache/helix/integration/controller/TestRedundantDroppedMessage.java
@@ -0,0 +1,72 @@
+
[3/3] helix git commit: Fix external view does not remove old ones
Fix external view does not remove old ones
Local cached external view may not contains the one that idealstate, current
states got removed. When controller switches, we shall read the external views
from Helix at least once.
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/73c3f0ad
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/73c3f0ad
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/73c3f0ad
Branch: refs/heads/master
Commit: 73c3f0ad81c4ef76cbd1e07d9368fe652099e56c
Parents: 0f0d417
Author: Junkai Xue
Authored: Mon Oct 29 16:13:48 2018 -0700
Committer: Junkai Xue
Committed: Thu Nov 1 17:10:46 2018 -0700
--
.../controller/stages/ClusterDataCache.java | 21 +---
1 file changed, 18 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/73c3f0ad/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
--
diff --git
a/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
b/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
index 08e98cc..9311ca2 100644
---
a/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
+++
b/helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java
@@ -87,8 +87,8 @@ public class ClusterDataCache extends AbstractDataCache {
private Map> _idealStateRuleMap;
private Map> _missingTopStateMap
= new HashMap<>();
private Map> _lastTopStateLocationMap = new
HashMap<>();
- private Map _targetExternalViewMap = new HashMap<>();
- private Map _externalViewMap = new HashMap<>();
+ private Map _targetExternalViewMap;
+ private Map _externalViewMap;
private Map>>
_disabledInstanceForPartitionMap = new HashMap<>();
private Set _disabledInstanceSet = new HashSet<>();
private String _eventId = "NO_ID";
@@ -261,6 +261,15 @@ public class ClusterDataCache extends AbstractDataCache {
updateDisabledInstances();
+if (_externalViewMap == null) {
+ _externalViewMap =
accessor.getChildValuesMap(accessor.keyBuilder().externalViews());
+}
+
+if (_clusterConfig.isTargetExternalViewEnabled() && _targetExternalViewMap
== null) {
+ _targetExternalViewMap =
+
accessor.getChildValuesMap(accessor.keyBuilder().targetExternalViews());
+}
+
long endTime = System.currentTimeMillis();
LogUtil.logInfo(LOG, _eventId,
"END: ClusterDataCache.refresh() for cluster " + getClusterName() + ",
took "
@@ -782,7 +791,7 @@ public class ClusterDataCache extends AbstractDataCache {
}
public ExternalView getTargetExternalView(String resourceName) {
-return _targetExternalViewMap.get(resourceName);
+return _targetExternalViewMap == null ? null :
_targetExternalViewMap.get(resourceName);
}
public void updateTargetExternalView(String resourceName, ExternalView
targetExternalView) {
@@ -794,6 +803,9 @@ public class ClusterDataCache extends AbstractDataCache {
* @return
*/
public Map getExternalViews() {
+if (_externalViewMap == null) {
+ return Collections.emptyMap();
+}
return Collections.unmodifiableMap(_externalViewMap);
}
@@ -802,6 +814,9 @@ public class ClusterDataCache extends AbstractDataCache {
* @param externalViews
*/
public void updateExternalViews(List externalViews) {
+if (_externalViewMap == null) {
+ _externalViewMap = new HashMap<>();
+}
for (ExternalView externalView : externalViews) {
_externalViewMap.put(externalView.getResourceName(), externalView);
}
[jira] [Commented] (HELIX-782) TASK: Make TaskDriver use ZKClient's create when creating workflows
[ https://issues.apache.org/jira/browse/HELIX-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672389#comment-16672389 ] Hudson commented on HELIX-782: -- FAILURE: Integrated in Jenkins build helix #1566 (See [https://builds.apache.org/job/helix/1566/]) [HELIX-782] TASK: Make TaskDriver use ZKClient's create when creating (narendly: rev 3844ad60034b029f3bbd916f629a7969117c1b26) * (edit) helix-core/src/main/java/org/apache/helix/task/TaskUtil.java * (edit) helix-core/src/main/java/org/apache/helix/task/TaskDriver.java * (add) helix-core/src/test/java/org/apache/helix/task/TestWorkflowCreation.java > TASK: Make TaskDriver use ZKClient's create when creating workflows > --- > > Key: HELIX-782 > URL: https://issues.apache.org/jira/browse/HELIX-782 > Project: Apache Helix > Issue Type: Improvement >Reporter: Hunter L >Assignee: Hunter L >Priority: Major > > TaskDriver should use create() but currently is using set(), which just > overwrites ZNodes that are in ZK. This is undesirable and we need to fix it, > especially in the wake of ZNode restructuring. > AC: > 1. Make TaskDriver use create() instead of set() > 2. Add an integration test: > TestWorkflowCreation:testWorkflowCreationNoDuplicates() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-784) TASK: Fix a bug in getExpiredJobs
[ https://issues.apache.org/jira/browse/HELIX-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672390#comment-16672390 ] Hudson commented on HELIX-784: -- FAILURE: Integrated in Jenkins build helix #1566 (See [https://builds.apache.org/job/helix/1566/]) [HELIX-784] TASK: Fix a bug in getExpiredJobs (narendly: rev befb1036f8d8be2729a800d3dde88fc1362a6489) * (delete) helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskPersistDataStage.java * (add) helix-core/src/test/java/org/apache/helix/controller/stages/TestTaskStage.java * (edit) helix-core/src/main/java/org/apache/helix/task/TaskUtil.java > TASK: Fix a bug in getExpiredJobs > - > > Key: HELIX-784 > URL: https://issues.apache.org/jira/browse/HELIX-784 > Project: Apache Helix > Issue Type: Improvement >Reporter: Hunter L >Assignee: Hunter L >Priority: Major > > getExpiredJobs(), when the job config is null, would just continue instead of > adding it to expiredJobs so that the job cleanup/purge would be re-tried. > This could possibly cause purge failures to leave a lot of jobs un-purged > with just the job config missing in ZK. This RB fixes this. > Changelist: > 1. Add the job name to expiredJobs if the job config does not exist in ZK > 2. Add a more detailed description in the error log > 3. Add an integration test for two task-related stages: TaskPersistDataStage > and TaskGarbageCollectionStage in TestTaskStage.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672412#comment-16672412 ] ASF GitHub Bot commented on HELIX-780: -- GitHub user zhan849 opened a pull request: https://github.com/apache/helix/pull/289 [HELIX-780] add task user content related api and added more tests - added get/add task user content rest api - consolidated rest api behavior: when getting/adding user content, if job/workflow does not exist, throw 404 - added more test cases You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhan849/helix harry/tf-rest-api Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #289 commit 18aa67b6d5c703e5b938b2f915f52a6ca856e889 Author: Harry Zhang Date: 2018-10-09T21:31:00Z [HELIX-780] add task user content related api and added more tests > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
helix git commit: [HELIX-780] add task user content related api and added more tests
Repository: helix
Updated Branches:
refs/heads/master 73c3f0ad8 -> 18aa67b6d
[HELIX-780] add task user content related api and added more tests
Project: http://git-wip-us.apache.org/repos/asf/helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/helix/commit/18aa67b6
Tree: http://git-wip-us.apache.org/repos/asf/helix/tree/18aa67b6
Diff: http://git-wip-us.apache.org/repos/asf/helix/diff/18aa67b6
Branch: refs/heads/master
Commit: 18aa67b6d5c703e5b938b2f915f52a6ca856e889
Parents: 73c3f0a
Author: Harry Zhang
Authored: Tue Oct 9 14:31:00 2018 -0700
Committer: Harry Zhang
Committed: Thu Nov 1 17:42:22 2018 -0700
--
.../server/resources/helix/JobAccessor.java | 11 +-
.../server/resources/helix/TaskAccessor.java| 115 +++
.../resources/helix/WorkflowAccessor.java | 10 +-
.../helix/rest/server/AbstractTestClass.java| 38 +++---
.../helix/rest/server/TestJobAccessor.java | 23 +++-
.../helix/rest/server/TestTaskAccessor.java | 83 +
.../helix/rest/server/TestWorkflowAccessor.java | 23 +++-
7 files changed, 278 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/helix/blob/18aa67b6/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
--
diff --git
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
index a984428..5548bd3 100644
---
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
+++
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java
@@ -181,7 +181,9 @@ public class JobAccessor extends AbstractHelixResource {
Map contentStore =
taskDriver.getJobUserContentMap(workflowName, jobName);
if (contentStore == null) {
-return JSONRepresentation(Collections.emptyMap());
+return notFound(String.format(
+"Unable to find content store. Workflow (%s) or Job (%s) does not
exist.",
+workflowName, jobName));
}
return JSONRepresentation(contentStore);
} catch (ZkNoNodeException e) {
@@ -193,7 +195,7 @@ public class JobAccessor extends AbstractHelixResource {
@POST
@Path("{jobName}/userContent")
- public Response updateWorkflowUserContent(
+ public Response updateJobUserContent(
@PathParam("clusterId") String clusterId,
@PathParam("workflowName") String workflowName,
@PathParam("jobName") String jobName,
@@ -225,6 +227,11 @@ public class JobAccessor extends AbstractHelixResource {
default:
return badRequest(String.format("Command \"%s\" is not supported!",
cmd));
}
+} catch (NullPointerException npe) {
+ // ZkCacheBasedDataAccessor would throw npe if workflow or job does not
exist
+ return notFound(String.format(
+ "Unable to find content store. Workflow (%s) or Job (%s) does not
exist.",
+ workflowName, jobName));
} catch (Exception e) {
_logger.error("Failed to update user content store", e);
return serverError(e);
http://git-wip-us.apache.org/repos/asf/helix/blob/18aa67b6/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/TaskAccessor.java
--
diff --git
a/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/TaskAccessor.java
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/TaskAccessor.java
new file mode 100644
index 000..79290e1
--- /dev/null
+++
b/helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/TaskAccessor.java
@@ -0,0 +1,115 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import javax.ws.rs.GET;
+import javax.ws.rs.POST;
+import javax.ws.r
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672463#comment-16672463 ] ASF GitHub Bot commented on HELIX-780: -- Github user asfgit closed the pull request at: https://github.com/apache/helix/pull/289 > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-780) Support get/add rest api for workflow/job/task user content
[ https://issues.apache.org/jira/browse/HELIX-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672466#comment-16672466 ] Hudson commented on HELIX-780: -- FAILURE: Integrated in Jenkins build helix #1567 (See [https://builds.apache.org/job/helix/1567/]) [HELIX-780] add task user content related api and added more tests (hrzhang: rev 18aa67b6d5c703e5b938b2f915f52a6ca856e889) * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/TestJobAccessor.java * (add) helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/TaskAccessor.java * (add) helix-rest/src/test/java/org/apache/helix/rest/server/TestTaskAccessor.java * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/TestWorkflowAccessor.java * (edit) helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/JobAccessor.java * (edit) helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/WorkflowAccessor.java * (edit) helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java > Support get/add rest api for workflow/job/task user content > --- > > Key: HELIX-780 > URL: https://issues.apache.org/jira/browse/HELIX-780 > Project: Apache Helix > Issue Type: Task >Reporter: Harry Zhang >Assignee: Harry Zhang >Priority: Major > > Need to support get/add rest api for workflow/job/task user content > AC: > * finish implementation > * test code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
