[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236149#comment-15236149 ] ASF GitHub Bot commented on HELIX-524: -- Github user hongbozeng closed the pull request at: https://github.com/apache/helix/pull/6 > add getProgress() to Task interface > --- > > Key: HELIX-524 > URL: https://issues.apache.org/jira/browse/HELIX-524 > Project: Apache Helix > Issue Type: Improvement > Components: helix-core >Affects Versions: 0.6.4 >Reporter: Hongbo Zeng > Fix For: 0.6.5 > > > Add a getProgress to the Task interface, this is very helpful for long > running tasks, from which we know the status of a task and see if it's > blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a > task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167167#comment-14167167 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58689390 Kishore from JIRA comments: ``` Yes, thats the idea. With helix-ipc this will be possible rt? This can be extended to write rebalancers that talk to nodes to get the high water mark to decide new master. what do you think? ``` Jason from JIRA comments: ``` Yes. Helix-IPC will do the job. We can also extend the idea to get high water mark, etc. Helix task frame exposes Task interface, so users are not implementing StateModel directly. How can we make this available to Task also? ``` add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167742#comment-14167742 ] Hongbo Zeng commented on HELIX-524: --- I was thinking about the TaskRunner gets the progress from a task and push the update somewhere (like Kafka). Anyways, this sounds a better plan, I will put something together for review. add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jira] [Commented] (HELIX-524) add getProgress() to Task interface
Yes, thats the idea. With helix-ipc this will be possible rt? This can be extended to write rebalancers that talk to nodes to get the high water mark to decide new master. what do you think? On Thu, Oct 9, 2014 at 3:28 PM, ASF GitHub Bot (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165886#comment-14165886 ] ASF GitHub Bot commented on HELIX-524: -- Github user zzhang5 commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58587902 If we are not using ZK to invoke these methods, are we opening some kind of end-point e.g. via Netty or JMX on each participant? add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164469#comment-14164469 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-58448716 Since this is an open source project, the real answer to this question is I don't know. We've made two public releases with this change, and so if there is anyone who has integrated or is planning to integrate with the task framework, they would need to made a change. If `ProgressReportingTask` handles the JMX reporting itself (it's an abstract class, not an interface), then `TaskRunner` wouldn't need to care about progress at all. And back to the bogus return value discussion, I think customers who don't have a good sense of progress shouldn't be required to change their integration just to implement `getProgress()`. add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156118#comment-14156118 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57588016 I like this in general, but I have a couple questions/comments: * This is an interface change. Is there a way we can provide this functionality without requiring that all existing task implementations be rewritten? * What is the persistence story (if any)? Who calls getProgress? add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[
https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156204#comment-14156204
]
ASF GitHub Bot commented on HELIX-524:
--
Github user brandtg commented on the pull request:
https://github.com/apache/helix/pull/6#issuecomment-57596146
For persistence story, what about putting a new health report:
`/STAND_ALONE/INSTANCES/{instanceName}/HEALTHREPORT/taskStatus`?
Another thing to consider is that adding `#getProgress` to the task
interface requires users to be able to intelligently report task status. I
suspect that in practice someone might be annoyed at this extra responsibility,
and provide dummy numbers (sounds stupid, but saw it before).
Maybe a better approach would be to try to monitor on things we know a
priori about task (e.g. lifetime) and provide tools to inspect ones that seem
stuck (e.g. task/partition-addressable stack trace)?
add getProgress() to Task interface
---
Key: HELIX-524
URL: https://issues.apache.org/jira/browse/HELIX-524
Project: Apache Helix
Issue Type: Improvement
Components: helix-core
Affects Versions: 0.6.4
Reporter: Hongbo Zeng
Fix For: 0.6.5
Add a getProgress to the Task interface, this is very helpful for long
running tasks, from which we know the status of a task and see if it's
blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a
task is finished
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156218#comment-14156218 ] ASF GitHub Bot commented on HELIX-524: -- Github user brandtg commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57597306 Wait, that stuff's gone now? Ok never mind. Is persistence of the state of a running task completely necessary? (Like would JMX suffice?) add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157419#comment-14157419 ] ASF GitHub Bot commented on HELIX-524: -- Github user dayzzz commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57728028 * About the interface change, IMHO, this is a nature part of the task interface. If the existing customers want to update, adding the support is encouraged. We can have another interface for monitoring the progress, but doesn't seem to be a nice design. * The task framework should call the progress interface and expose the status somehow, discussed below. * Persistency story, ZK is a good place if we only want to record the final result. If we want to expose the progress as a task runs, putting these periodical status updates in ZK is not a choice due to the large traffic, generally ZK is not a good place for reporting and monitoring service status. I also discussed this with Jason, we thought about inGraph (which is not an option for open source), Kafka or Riemann. (Greg, JMX sounds a good idea.) Without a conclusion of where to put these status stats, I agree that the progress interface is not of much value. For the first step, it would be good enough if we can monitor the progress. What do you guys think? * For the bogus progress number, it's the customer themselves who need to track the progress, if they want to see the bogus value, I'm fine with that :). The controller should be set not to act on the bogus value by the customers. add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157483#comment-14157483 ] ASF GitHub Bot commented on HELIX-524: -- Github user kanakb commented on the pull request: https://github.com/apache/helix/pull/6#issuecomment-57730714 Why not have an abstract ProgressReportingTask that implements task and includes all the JMX persistence, and then have tasks extends from that and implement getProgress()? That avoids the interface change, but also allows smarter tasks to let their progress be known. add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154757#comment-14154757 ] A Sebastian commented on HELIX-524: --- How do you expect this method to be used by the framework? add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155602#comment-14155602 ] Hongbo Zeng commented on HELIX-524: --- 1. For monitoring purpose. The framework can check the progress periodically and reflect it in Helix, e.g. in PropertyStore, such that users can check it. In a distributed system, there are always outliers running slowly, customers need to know which ones are slow and how slow they are. Especially helpful for identifying tasks which get stuck. 2. Take action on outlier tasks. The framework can do something smart if it finds tasks that are stuck or running slowly. It can optionally cancel the task and restart it, maybe somewhere else. Customers can also take some actions on those. add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HELIX-524) add getProgress() to Task interface
[ https://issues.apache.org/jira/browse/HELIX-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155861#comment-14155861 ] ASF GitHub Bot commented on HELIX-524: -- GitHub user dayzzz opened a pull request: https://github.com/apache/helix/pull/6 [HELIX-524] Add a getProgress to the Task interface [HELIX-524] Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished You can merge this pull request into a Git repository by running: $ git pull https://github.com/dayzzz/helix master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/6.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6 commit 5d1b27f81abc4c67301d864f122e82f5a0ce49c3 Author: Hongbo Zeng hz...@linkedin.com Date: 2014-10-02T00:19:49Z [HELIX-524] Add a getProgress to the Task interface add getProgress() to Task interface --- Key: HELIX-524 URL: https://issues.apache.org/jira/browse/HELIX-524 Project: Apache Helix Issue Type: Improvement Components: helix-core Affects Versions: 0.6.4 Reporter: Hongbo Zeng Fix For: 0.6.5 Add a getProgress to the Task interface, this is very helpful for long running tasks, from which we know the status of a task and see if it's blocked. The return value is a double, ranging from 0 to 1.0, 1.0 indicates a task is finished -- This message was sent by Atlassian JIRA (v6.3.4#6332)
