[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242051#comment-16242051
 ] 

ASF GitHub Bot commented on FLINK-7648:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4824


> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212573#comment-16212573
 ] 

ASF GitHub Bot commented on FLINK-7648:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4824
  
Thanks for your review @zentol. Once the preceding PR are merged, I'll 
merge this one as well.


> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207654#comment-16207654
 ] 

ASF GitHub Bot commented on FLINK-7648:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4824
  
The handler related commits look good to me.


> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203642#comment-16203642
 ] 

ASF GitHub Bot commented on FLINK-7648:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4824

[FLINK-7648] [flip6] Add TaskManagersHandler

## What is the purpose of the change

Add the `TaskManagersHandler` for the new REST server endpoint.

## Brief change log

- Added `TaskManagersHandler` and `TaskManagersHeaders` for serving 
`TaskManagersInfo`
- Changed the serialization of the `legacy.TaskManagersHandler` to reflect 
`TaskManagerInfo` serialization where the `HardwareDescription` is stored in a 
nested JSON object.

## Verifying this change

Added tests:

- `TaskManagerInfoTest` and `TaskManagersInfoTest` for (un)marshalling

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink addTaskManagersHandler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4824


commit 77ece83513d9a79f2f0c3e7024d91243026c334c
Author: Till 
Date:   2017-10-13T07:19:51Z

[FLINK-7832] [flip6] Extend SlotManager to report free slots per TM

Fail if slot belongs to a unregistered TaskManager

Add more sanity checks

Make the TaskManagerSlot state transitions clearer

Introduce proper TaskManagerSlot state enum

Refactor SlotManager for better maintainability

commit 72ae21be368fc3de32a12e486c73a4fb69a76abf
Author: Till 
Date:   2017-10-12T16:24:02Z

[FLINK-7831] Make last received heartbeat retrievable

This commit adds functionality to retrieve the last received heartbeat from
the HeartbeatManager.

commit fceae2d4fa1bda55f2f9b412388606747e9b24af
Author: Till Rohrmann 
Date:   2017-10-11T22:40:17Z

[FLINK-7648] [flip6] Add TaskManagersHandler

Send dataPort and HardwareDescription to RM

Instantiate RM leader retriever

commit c7f637da68d9627b6dcc73ac4a88d278d7b4b5b5
Author: Till 
Date:   2017-10-13T14:32:56Z

[FLINK-7648] [flip6] Write HardwareDescription in sub object of 
legacy.TaskManagersHandler




> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187834#comment-16187834
 ] 

Till Rohrmann commented on FLINK-7648:
--

Have you started the test with the {{flink-runtime-web}} dependencies? 
Otherwise there are no web ui files. You can also query your server using curl 
to see whether it produces the right output.

Currently, there is no easy way to start a job, because the client is currently 
being worked on. See Flink-7072 for more information. The best way to test the 
code would be in a unit test right now.

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186631#comment-16186631
 ] 

Bowen Li commented on FLINK-7648:
-

[~till.rohrmann] I successfully started the cluster in a unit test. But, the 
browser shows {"errors":["Not found."]} when I visited 
{{http://127.0.0.1:9067/}}, and there's no UI. Is this expected, or did I do 
anything wrong? 

BTW, how to run a job in such a {{StandaloneSessionCluster}}. I'm porting 
{{JobMetricsHandler}} and it requires a running job to be able to visit that UI 
component.

Sorry for the back-and-forth questionsI'm still learning about the new 
server architecture

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185502#comment-16185502
 ] 

Till Rohrmann commented on FLINK-7648:
--

In order to wait on the completion you can do

{code}
@Test
public void testStandaloneSessionCluster() throws ExecutionException, 
InterruptedException {
Configuration configuration = new Configuration();
configuration.setString(JobManagerOptions.ADDRESS, "localhost");

StandaloneSessionClusterEntrypoint standaloneSessionClusterEntrypoint = 
new StandaloneSessionClusterEntrypoint(configuration);

standaloneSessionClusterEntrypoint.startCluster();

standaloneSessionClusterEntrypoint.getTerminationFuture().get();
}
{code}

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185433#comment-16185433
 ] 

Bowen Li commented on FLINK-7648:
-

[~till.rohrmann] What do you mean by "wait on its completion via the 
termination future"?

I wrote the following test, and it immediately terminates:

{code:java}
public class Test {
public static void main(String[] args) {
String configDir = 
"flink-dist/target/flink-1.4-SNAPSHOT-bin/flink-1.4-SNAPSHOT/conf";
String[] arg = {"--configDir", configDir};
StandaloneSessionClusterEntrypoint.main(arg);
}
}
{code}

```

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182234#comment-16182234
 ] 

Till Rohrmann commented on FLINK-7648:
--

[~phoenixjiangnan], currently the new REST server endpoint runs under port 
{{9067}}. Even easier than building Flink all the time is to write a test which 
starts the {{StandaloneSessionClusterEntrypoint}} and then wait on its 
completion via the termination future (this should never terminate in fact). If 
you now start the test with {{flink-runtime-web}} module dependencies (under 
settings of the test), then it will start a rest server endpoint which can also 
serve the web gui.

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181315#comment-16181315
 ] 

Bowen Li commented on FLINK-7648:
-

[~till.rohrmann] Hi Till, I built Flink from source, copied 
flink-runtime-web-xxx.jar to {{build/lib}} (so flink-runtime-web is on 
classpath), and ran {{bin/start-cluster.sh flip6}}, but I couldn't connect to 
http://localhost:8081 from my browser. Did I miss anything?

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180540#comment-16180540
 ] 

Till Rohrmann commented on FLINK-7648:
--

Yes exactly, we have to port all four of them.

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180437#comment-16180437
 ] 

Bowen Li commented on FLINK-7648:
-

[~till.rohrmann] Thanks Till. Yeah, that sounds too complicated as a starting 
point for me. Feel free to assign this ticket to others if you don't hear from 
me.

For metrics handlers, do you mean that we need to port all subclasses of 
{{AbstractMetricsHandler}} to new REST endpoint? I found there are four 
subclasses, so I probably can migrate one of them first?

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180418#comment-16180418
 ] 

Till Rohrmann commented on FLINK-7648:
--

Hi [~phoenixjiangnan], this one is indeed one of the more complex handlers, 
because there are still things missing on the {{ResourceManager}} side.

In a nutshell: 

- We have to register the {{HardwareInformation}} for each {{TaskExecutor}} at 
the {{ResourceManager}} to answer the request
- We have to add a {{LeaderGatewayRetriever}} to this handler 
to be able to talk to the {{ResourceManager}}

You can test the web gui by either starting a Flip-6 local cluster via 
`bin/start-cluster.sh flip6` or by starting the 
`StandaloneSessionClusterEntrypoint` with `flink-runtime-web` in its classpath.

Testing the handler itself should be quite simple. Just write a junit test. The 
integration would be a bit more involved by spawning a RM which can serve the 
TM information.

If this should be too involved, then I would recommend starting with the 
metrics handlers ({{AbstractMetricsHandler}}).

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7648) Port TaskManagersHandler to new REST endpoint

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180227#comment-16180227
 ] 

Bowen Li commented on FLINK-7648:
-

[~till.rohrmann] [~tzulitai] I'd like to try this and contribute to the new 
server architecture.

Given it's a handler for TaskManager, I guess I chose a pretty hard one... :) 
Can you please give me some guidance on this ticket, e.g.

- how to start and run the new web portal?
- what's the expected way to test the endpoint?
- more

Thanks!

> Port TaskManagersHandler to new REST endpoint
> -
>
> Key: FLINK-7648
> URL: https://issues.apache.org/jira/browse/FLINK-7648
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Bowen Li
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{TaskManagersHandler}} to the new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)