[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236149#comment-15236149 ] ASF GitHub Bot commented on HELIX-524: -- Github user hongbozeng closed the pull request at: https://github.com/apache/helix/pull/6 > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167742#comment-14167742 ] Hongbo Zeng commented on HELIX-524: --- I was thinking about the TaskRunner gets the progress from a task and push the update somewhere (like Kafka). Anyways, this sounds a better plan, I will put something together for review. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167167#comment-14167167 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58689390 Kishore from JIRA comments: ``` Yes, thats the idea. With helix-ipc this will be possible rt? This can be extended to write rebalancers that talk to nodes to get the high water mark to decide new master. what do you think? ``` Jason from JIRA comments: ``` Yes. Helix-IPC will do the job. We can also extend the idea to get high water mark, etc. Helix task frame exposes Task interface, so users are not implementing StateModel directly. How can we make this available to Task also? ``` > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165886#comment-14165886 ] ASF GitHub Bot commented on HELIX-524: -- Github user zzhang5 commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58587902 If we are not using ZK to invoke these methods, are we opening some kind of end-point e.g. via Netty or JMX on each participant? > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164661#comment-14164661 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58459614 I think having a `@ProgressReporter` could be an elegant way to allow users to plug in additional functionality. Progress could be just the first thing a task could expose, and in the future if we think of others, we just need to add a new annotation. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[
https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164660#comment-14164660
]
ASF GitHub Bot commented on HELIX-524:
--
Github user kanakb commented on the pull request:
https://github.com/apache/helix/pull/6#issuecomment-58459371
Email from Kishore:
```
Apologize to throw in another idea into the mix. Why not have additional
methods in the implementation that the agent can invoke on demand based on
annotation?. This extends the way we invoke methods on the statemodel.
For example we have
@Transition(TO=MASTER, FROM=SLAVE)
void fromSlaveTOMaster(Message m, NotificationContext ctx){
}
Can we have something similar as
@Method(name="getProgress")
Response getProgress(Message m, NotificationContext ctx){
}
Helix provides a generic way to invoke a method on a partition. I think
this is more powerful , does not disturb any interfaces and can be extending to
do custom stuff.
Also these methods will not be invoked via ZK, instead the controller can
directly invoke the method on the participant.
Feedback?
```
> add getProgress() to Task interface
> ---
>
> Key: HELIX-524
> URL: https://issues.apache.org/jira/browse/HELIX-524
> Project: Apache Helix
> Issue Type: Improvement
> Components: helix-core
>Affects Versions: 0.6.4
>Reporter: Hongbo Zeng
> Fix For: 0.6.5
>
>
> Add a getProgress to the Task interface, this is very helpful for long
> running tasks, from which we know the status of a task and see if it's
> blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a
> task is finished
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[
https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164495#comment-14164495
]
kishore gopalakrishna commented on HELIX-524:
-
Apologize to throw in another idea into the mix. Why not have additional
methods in the implementation that the agent can invoke on demand based on
annotation?. This extends the way we invoke methods on the statemodel.
For example we have
@Transition(TO=MASTER, FROM=SLAVE)
void fromSlaveTOMaster(Message m, NotificationContext ctx){
}
Can we have something similar as
@Method(name="getProgress")
Response getProgress(Message m, NotificationContext ctx){
}
Helix provides a generic way to invoke a method on a partition. I think this is
more powerful , does not disturb any interfaces and can be extending to do
custom stuff.
Also these methods will not be invoked via ZK, instead the controller can
directly invoke the method on the participant.
Feedback?
> add getProgress() to Task interface
> ---
>
> Key: HELIX-524
> URL: https://issues.apache.org/jira/browse/HELIX-524
> Project: Apache Helix
> Issue Type: Improvement
> Components: helix-core
>Affects Versions: 0.6.4
>Reporter: Hongbo Zeng
> Fix For: 0.6.5
>
>
> Add a getProgress to the Task interface, this is very helpful for long
> running tasks, from which we know the status of a task and see if it's
> blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a
> task is finished
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164469#comment-14164469 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58448716 Since this is an open source project, the real answer to this question is "I don't know." We've made two public releases with this change, and so if there is anyone who has integrated or is planning to integrate with the task framework, they would need to made a change. If `ProgressReportingTask` handles the JMX reporting itself (it's an abstract class, not an interface), then `TaskRunner` wouldn't need to care about progress at all. And back to the "bogus" return value discussion, I think customers who don't have a good sense of progress shouldn't be required to change their integration just to implement `getProgress()`. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164230#comment-14164230 ] ASF GitHub Bot commented on HELIX-524: -- Github user dayzzz commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58433144 Sorry for the delay. Are there any customers outside the Espresso team? If we are the only customer right now, it would not be too costly to update the implementation with the interface. Making the getProgress into the interface has a good thing that it naturally goes into TaskRunner and we don't need to do something like "instance of ProgressReportingTask" and then call the getProgress, or a subclass of TaskRunner which is specific for ProgressReportingTask and call the getProgress. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157483#comment-14157483 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57730714 Why not have an abstract ProgressReportingTask that implements task and includes all the JMX persistence, and then have tasks extends from that and implement getProgress()? That avoids the interface change, but also allows "smarter" tasks to let their progress be known. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157419#comment-14157419 ] ASF GitHub Bot commented on HELIX-524: -- Github user dayzzz commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57728028 * About the interface change, IMHO, this is a nature part of the task interface. If the existing customers want to update, adding the support is encouraged. We can have another interface for monitoring the progress, but doesn't seem to be a nice design. * The task framework should call the progress interface and expose the status somehow, discussed below. * Persistency story, ZK is a good place if we only want to record the final result. If we want to expose the progress as a task runs, putting these periodical status updates in ZK is not a choice due to the large traffic, generally ZK is not a good place for reporting and monitoring service status. I also discussed this with Jason, we thought about inGraph (which is not an option for open source), Kafka or Riemann. (Greg, JMX sounds a good idea.) Without a conclusion of where to put these status stats, I agree that the progress interface is not of much value. For the first step, it would be good enough if we can monitor the progress. What do you guys think? * For the bogus progress number, it's the customer themselves who need to track the progress, if they want to see the bogus value, I'm fine with that :). The controller should be set not to act on the bogus value by the customers. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156218#comment-14156218 ] ASF GitHub Bot commented on HELIX-524: -- Github user brandtg commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57597306 Wait, that stuff's gone now? Ok never mind. Is persistence of the state of a running task completely necessary? (Like would JMX suffice?) > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[
https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156204#comment-14156204
]
ASF GitHub Bot commented on HELIX-524:
--
Github user brandtg commented on the pull request:
https://github.com/apache/helix/pull/6#issuecomment-57596146
For persistence story, what about putting a new health report:
`/STAND_ALONE/INSTANCES/{instanceName}/HEALTHREPORT/taskStatus`?
Another thing to consider is that adding `#getProgress` to the task
interface requires users to be able to intelligently report task status. I
suspect that in practice someone might be annoyed at this extra responsibility,
and provide dummy numbers (sounds stupid, but saw it before).
Maybe a better approach would be to try to monitor on things we know a
priori about task (e.g. lifetime) and provide tools to inspect ones that seem
stuck (e.g. task/partition-addressable stack trace)?
> add getProgress() to Task interface
> ---
>
> Key: HELIX-524
> URL: https://issues.apache.org/jira/browse/HELIX-524
> Project: Apache Helix
> Issue Type: Improvement
> Components: helix-core
>Affects Versions: 0.6.4
>Reporter: Hongbo Zeng
> Fix For: 0.6.5
>
>
> Add a getProgress to the Task interface, this is very helpful for long
> running tasks, from which we know the status of a task and see if it's
> blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a
> task is finished
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156118#comment-14156118 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57588016 I like this in general, but I have a couple questions/comments: * This is an interface change. Is there a way we can provide this functionality without requiring that all existing task implementations be rewritten? * What is the persistence story (if any)? Who calls getProgress? > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155861#comment-14155861 ] ASF GitHub Bot commented on HELIX-524: -- GitHub user dayzzz opened a pull request: https://github.com/apache/helix/pull/6 [HELIX-524] Add a getProgress to the Task interface [HELIX-524] Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished You can merge this pull request into a Git repository by running: $ git pull https://github.com/dayzzz/helix master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/6.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6 commit 5d1b27f81abc4c67301d864f122e82f5a0ce49c3 Author: Hongbo Zeng Date: 2014-10-02T00:19:49Z [HELIX-524] Add a getProgress to the Task interface > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155602#comment-14155602 ] Hongbo Zeng commented on HELIX-524: --- 1. For monitoring purpose. The framework can check the progress periodically and reflect it in Helix, e.g. in PropertyStore, such that users can check it. In a distributed system, there are always outliers running slowly, customers need to know which ones are slow and how slow they are. Especially helpful for identifying tasks which get stuck. 2. Take action on outlier tasks. The framework can do something smart if it finds tasks that are stuck or running slowly. It can optionally cancel the task and restart it, maybe somewhere else. Customers can also take some actions on those. > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154757#comment-14154757 ] A Sebastian commented on HELIX-524: --- How do you expect this method to be used by the framework? > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
