[jira] [Commented] (FLINK-4343) Implement new TaskManager

2016-10-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579610#comment-15579610
 ] 

ASF GitHub Bot commented on FLINK-4343:
---

Github user tillrohrmann closed the pull request at:

https://github.com/apache/flink/pull/2640


> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
>  Issue Type: New Feature
>  Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}} 
> changes based on FLIP-6 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the 
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the 
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current 
> implementation, we will drop the old {{TaskManager}} and rename the 
> {{TaskExecutor}} to {{TaskManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FLINK-4343) Implement new TaskManager

2016-10-16 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579608#comment-15579608
 ] 

ASF GitHub Bot commented on FLINK-4343:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2640
  
Failing test case is unrelated. Will merge the PR into the flip-6 branch.


> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
>  Issue Type: New Feature
>  Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}} 
> changes based on FLIP-6 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the 
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the 
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current 
> implementation, we will drop the old {{TaskManager}} and rename the 
> {{TaskExecutor}} to {{TaskManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FLINK-4343) Implement new TaskManager

2016-10-15 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577967#comment-15577967
 ] 

ASF GitHub Bot commented on FLINK-4343:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2640

[FLINK-4343] [tm] Implement TM -> JM registration logic

Upon requesting a slot for a new job, the TaskManager registers this job at 
the
`JobLeaderService`. The job leader service is responsible to monitor job 
leader changes
for all registered jobs. In case of a new job leader, the service will try 
to establish
a connection to the new job leader. Upon establishing the connection the 
task manager
is informed about it. The task manager will then offer all allocated but 
not yet active
slots to the new job leader.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink taskManagerRegistration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2640.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2640


commit aafd9020eae2769fd0ade9d922683aac2886412a
Author: Till Rohrmann 
Date:   2016-10-05T15:02:06Z

[FLINK-4343] [tm] Implement TM -> JM registration logic

Upon requesting a slot for a new job, the TaskManager registers this job at 
the
JobLeaderService. The job leader service is responsible to monitor job 
leader changes
for all registered jobs. In case of a new job leader, the service will try 
to establish
a connection to the new job leader. Upon establishing the connection the 
task manager
is informed about it. The task manager will then offer all allocated but 
not yet active
slots to the new job leader.

Implement JobLeaderService

The JobLeaderService is responsible for establishing a connection to the JM 
leader of a given
job.

Disable TaskExecutorTest#testRejectAllocationRequestsForOutOfSyncSlots

Add simple task submission test

Add job leader detection test case

Add task slot acceptance test

Fix RpcCompletenessTest

Add comments




> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
>  Issue Type: New Feature
>  Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}} 
> changes based on FLIP-6 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the 
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the 
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current 
> implementation, we will drop the old {{TaskManager}} and rename the 
> {{TaskExecutor}} to {{TaskManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FLINK-4343) Implement new TaskManager

2016-08-25 Thread Till Rohrmann (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436982#comment-15436982
 ] 

Till Rohrmann commented on FLINK-4343:
--

I've tried to outline the interaction of the new {{TaskManager}} with the 
{{ResourceManager}} and the {{JobManager}}. I've attached the document as a 
link to this issue. The main addition compared to the existing {{TaskManager}} 
would be the introduction of a {{SlotManager}} which manages the 
slot-JobManager assignment.

> Implement new TaskManager
> -
>
> Key: FLINK-4343
> URL: https://issues.apache.org/jira/browse/FLINK-4343
> Project: Flink
>  Issue Type: New Feature
>  Components: Cluster Management
>Reporter: Stephan Ewen
>
> This is the parent issue for the efforts to implement the {{TaskManager}} 
> changes based on FLIP-6 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
> Because of the breadth of changes, we should implement a new version of the 
> {{TaskManager}} (let's call it {{TaskExecutor}}) rather than updating the 
> current {{TaskManager}}. That will allow us to keep a working master branch.
> At the point when the new cluster management is on par with the current 
> implementation, we will drop the old {{TaskManager}} and rename the 
> {{TaskExecutor}} to {{TaskManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)