Re: Two issues when deploying Flink on DC/OS

2018-02-09 Thread Till Rohrmann 
Hi,

"java.io.IOException: Connection reset by peer" is usually thrown if the
remote peer terminates the connection. So the interesting bit would be
who's requesting static files from Flink. So far we serve the web frontend
and the log and stdout files via the StaticFileServerHandler. Maybe it's
DC/OS itself for some reason? This error should, however, not influence
Flink itself.

Cheers,
Till

On Thu, Feb 8, 2018 at 12:36 PM, Lasse Nedergaard  wrote:

> And We see the same too
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
> Den 8. feb. 2018 kl. 11.58 skrev Stavros Kontopoulos <
> st.kontopou...@gmail.com>:
>
> We see the same issue here (2):
> 2018-02-08 10:55:11,447 ERROR 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler
>  - Caught exception
> java.io.IOException: Connection reset by peer
>
> Stavros
>
> On Sat, Jan 13, 2018 at 9:59 PM, Eron Wright  wrote:
>
>> Hello Dongwon,
>>
>> Flink doesn't support a 'unique host' constraint at this time; it simply
>> accepts adequate offers without any such consideration.   Flink does
>> support a 'host attributes' constraint to filter certain hosts, but that's
>> not applicable here.
>>
>> Under the hood, Flink uses a library called Netflix Fenzo to optimize
>> placement, and a uniqueness constraint could be added by more deeply
>> leveraging Fenzo's constraint system.   You mentioned that you're trying to
>> make good use of your GPU resources, which could also be achieved by
>> treating GPU as a scalar resource (similar to how memory and cores are
>> treated).   Mesos does support that, but Fenzo may require some
>> enhancement.   So, these are two potential ways to enhance Flink to support
>> your scenario.  I'm happy to help; reach out to me.
>>
>> The obvious, ugly workaround is to configure your TMs to be large enough
>> to consume the whole host.
>>
>> Eron
>>
>>
>>
>>
>>
>> On Thu, Jan 11, 2018 at 7:18 AM, Gary Yao  wrote:
>>
>>> Hi Dongwon,
>>>
>>> I am not familiar with the deployment on DC/OS. However, Eron Wright and
>>> Jörg
>>> Schad (cc'd), who have worked on the Mesos integration, might be able to
>>> help
>>> you.
>>>
>>> Best,
>>> Gary
>>>
>>> On Tue, Jan 9, 2018 at 10:29 AM, Dongwon Kim 
>>> wrote:
>>>
 Hi,

 I've launched JobManager and TaskManager on DC/OS successfully.
 Now I have two new issues:

 1) All TaskManagers are scheduled on a single node.
 - Is it intended to maximize data locality and minimize network
 communication cost?
 - Is there an option in Flink to adjust the behavior of JobManager when
 it considers multiple resource offers from different Mesos agents?
 - I want to schedule TaskManager processes on different GPU servers so
 that each TaskManger process can use its own GPU cards exclusively.
 - Below is a part of JobManager log that is occurring while JobManager
 is negotiating resources with the Mesos master:

 2018-01-09 07:34:54,872 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosJobManager  - 
 JobManager akka.tcp://flink@dnn-g08-233:18026/user/jobmanager was granted 
 leadership with leader session ID 
 Some(----).
 2018-01-09 07:34:55,889 INFO  
 org.apache.flink.mesos.scheduler.ConnectionMonitor- Connecting 
 to Mesos...
 2018-01-09 07:34:55,962 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Trying to associate with JobManager leader 
 akka.tcp://flink@dnn-g08-233:18026/user/jobmanager
 2018-01-09 07:34:55,977 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Resource Manager associating with leading JobManager 
 Actor[akka://flink/user/jobmanager#-1481183359] - leader session 
 ----
 2018-01-09 07:34:56,479 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-1 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-2 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-3 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-4 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-5 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,483 INFO

Re: Two issues when deploying Flink on DC/OS

2018-02-08 Thread Lasse Nedergaard 
And We see the same too

Med venlig hilsen / Best regards
Lasse Nedergaard


> Den 8. feb. 2018 kl. 11.58 skrev Stavros Kontopoulos 
> :
> 
> We see the same issue here (2):
> 2018-02-08 10:55:11,447 ERROR 
> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler  - Caught 
> exception
> java.io.IOException: Connection reset by peer
> 
> Stavros
> 
>> On Sat, Jan 13, 2018 at 9:59 PM, Eron Wright  wrote:
>> Hello Dongwon,
>> 
>> Flink doesn't support a 'unique host' constraint at this time; it simply 
>> accepts adequate offers without any such consideration.   Flink does support 
>> a 'host attributes' constraint to filter certain hosts, but that's not 
>> applicable here.
>> 
>> Under the hood, Flink uses a library called Netflix Fenzo to optimize 
>> placement, and a uniqueness constraint could be added by more deeply 
>> leveraging Fenzo's constraint system.   You mentioned that you're trying to 
>> make good use of your GPU resources, which could also be achieved by 
>> treating GPU as a scalar resource (similar to how memory and cores are 
>> treated).   Mesos does support that, but Fenzo may require some enhancement. 
>>   So, these are two potential ways to enhance Flink to support your 
>> scenario.  I'm happy to help; reach out to me.
>> 
>> The obvious, ugly workaround is to configure your TMs to be large enough to 
>> consume the whole host.
>> 
>> Eron
>> 
>> 
>> 
>> 
>> 
>>> On Thu, Jan 11, 2018 at 7:18 AM, Gary Yao  wrote:
>>> Hi Dongwon,
>>> 
>>> I am not familiar with the deployment on DC/OS. However, Eron Wright and 
>>> Jörg
>>> Schad (cc'd), who have worked on the Mesos integration, might be able to 
>>> help
>>> you.
>>> 
>>> Best,
>>> Gary
>>> 
 On Tue, Jan 9, 2018 at 10:29 AM, Dongwon Kim  wrote:
 Hi,
 
 I've launched JobManager and TaskManager on DC/OS successfully.
 Now I have two new issues:
 
 1) All TaskManagers are scheduled on a single node. 
 - Is it intended to maximize data locality and minimize network 
 communication cost?
 - Is there an option in Flink to adjust the behavior of JobManager when it 
 considers multiple resource offers from different Mesos agents?
 - I want to schedule TaskManager processes on different GPU servers so 
 that each TaskManger process can use its own GPU cards exclusively.  
 - Below is a part of JobManager log that is occurring while JobManager is 
 negotiating resources with the Mesos master:
 2018-01-09 07:34:54,872 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosJobManager  - 
 JobManager akka.tcp://flink@dnn-g08-233:18026/user/jobmanager was granted 
 leadership with leader session ID 
 Some(----).
 2018-01-09 07:34:55,889 INFO  
 org.apache.flink.mesos.scheduler.ConnectionMonitor- Connecting 
 to Mesos...
 2018-01-09 07:34:55,962 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Trying to associate with JobManager leader 
 akka.tcp://flink@dnn-g08-233:18026/user/jobmanager
 2018-01-09 07:34:55,977 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Resource Manager associating with leading JobManager 
 Actor[akka://flink/user/jobmanager#-1481183359] - leader session 
 ----
 2018-01-09 07:34:56,479 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-1 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-2 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-3 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-4 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,481 INFO  
 org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
 - Scheduling Mesos task taskmanager-5 with (10240.0 MB, 8.0 cpus).
 2018-01-09 07:34:56,483 INFO  
 org.apache.flink.mesos.scheduler.LaunchCoordinator- Now 
 gathering offers for at least 5 task(s).
 2018-01-09 07:34:56,484 INFO  
 org.apache.flink.mesos.scheduler.ConnectionMonitor- Connected 
 to Mesos as framework ID 59b85b42-a4a2-4632-9578-9e480585ecdc-0004.
 2018-01-09 07:34:56,690 INFO  
 org.apache.flink.mesos.scheduler.LaunchCoordinator- Received 
 offer(s) of 606170.0 MB, 234.2 cpus:
 2018-01-09 07:34:56,692 INFO

Re: Two issues when deploying Flink on DC/OS

2018-02-08 Thread Stavros Kontopoulos 
We see the same issue here (2):
2018-02-08 10:55:11,447 ERROR
org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler  - Caught
exception
java.io.IOException: Connection reset by peer

Stavros

On Sat, Jan 13, 2018 at 9:59 PM, Eron Wright  wrote:

> Hello Dongwon,
>
> Flink doesn't support a 'unique host' constraint at this time; it simply
> accepts adequate offers without any such consideration.   Flink does
> support a 'host attributes' constraint to filter certain hosts, but that's
> not applicable here.
>
> Under the hood, Flink uses a library called Netflix Fenzo to optimize
> placement, and a uniqueness constraint could be added by more deeply
> leveraging Fenzo's constraint system.   You mentioned that you're trying to
> make good use of your GPU resources, which could also be achieved by
> treating GPU as a scalar resource (similar to how memory and cores are
> treated).   Mesos does support that, but Fenzo may require some
> enhancement.   So, these are two potential ways to enhance Flink to support
> your scenario.  I'm happy to help; reach out to me.
>
> The obvious, ugly workaround is to configure your TMs to be large enough
> to consume the whole host.
>
> Eron
>
>
>
>
>
> On Thu, Jan 11, 2018 at 7:18 AM, Gary Yao  wrote:
>
>> Hi Dongwon,
>>
>> I am not familiar with the deployment on DC/OS. However, Eron Wright and
>> Jörg
>> Schad (cc'd), who have worked on the Mesos integration, might be able to
>> help
>> you.
>>
>> Best,
>> Gary
>>
>> On Tue, Jan 9, 2018 at 10:29 AM, Dongwon Kim 
>> wrote:
>>
>>> Hi,
>>>
>>> I've launched JobManager and TaskManager on DC/OS successfully.
>>> Now I have two new issues:
>>>
>>> 1) All TaskManagers are scheduled on a single node.
>>> - Is it intended to maximize data locality and minimize network
>>> communication cost?
>>> - Is there an option in Flink to adjust the behavior of JobManager when
>>> it considers multiple resource offers from different Mesos agents?
>>> - I want to schedule TaskManager processes on different GPU servers so
>>> that each TaskManger process can use its own GPU cards exclusively.
>>> - Below is a part of JobManager log that is occurring while JobManager
>>> is negotiating resources with the Mesos master:
>>>
>>> 2018-01-09 07:34:54,872 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosJobManager  - 
>>> JobManager akka.tcp://flink@dnn-g08-233:18026/user/jobmanager was granted 
>>> leadership with leader session ID 
>>> Some(----).
>>> 2018-01-09 07:34:55,889 INFO  
>>> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connecting 
>>> to Mesos...
>>> 2018-01-09 07:34:55,962 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Trying to associate with JobManager leader 
>>> akka.tcp://flink@dnn-g08-233:18026/user/jobmanager
>>> 2018-01-09 07:34:55,977 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Resource Manager associating with leading JobManager 
>>> Actor[akka://flink/user/jobmanager#-1481183359] - leader session 
>>> ----
>>> 2018-01-09 07:34:56,479 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Scheduling Mesos task taskmanager-1 with (10240.0 MB, 8.0 cpus).
>>> 2018-01-09 07:34:56,481 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Scheduling Mesos task taskmanager-2 with (10240.0 MB, 8.0 cpus).
>>> 2018-01-09 07:34:56,481 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Scheduling Mesos task taskmanager-3 with (10240.0 MB, 8.0 cpus).
>>> 2018-01-09 07:34:56,481 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Scheduling Mesos task taskmanager-4 with (10240.0 MB, 8.0 cpus).
>>> 2018-01-09 07:34:56,481 INFO  
>>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  
>>> - Scheduling Mesos task taskmanager-5 with (10240.0 MB, 8.0 cpus).
>>> 2018-01-09 07:34:56,483 INFO  
>>> org.apache.flink.mesos.scheduler.LaunchCoordinator- Now 
>>> gathering offers for at least 5 task(s).
>>> 2018-01-09 07:34:56,484 INFO  
>>> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connected 
>>> to Mesos as framework ID 59b85b42-a4a2-4632-9578-9e480585ecdc-0004.
>>> 2018-01-09 07:34:56,690 INFO  
>>> org.apache.flink.mesos.scheduler.LaunchCoordinator- Received 
>>> offer(s) of 606170.0 MB, 234.2 cpus:
>>> 2018-01-09 07:34:56,692 INFO  
>>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>>> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2174 from 50.1.100.233 of 86.0 
>>> MB, 45.9 cpus for [*]
>>> 2018-01-09 07:34:56,692 INFO  
>>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>>>

Re: Two issues when deploying Flink on DC/OS

2018-01-13 Thread Eron Wright 
Hello Dongwon,

Flink doesn't support a 'unique host' constraint at this time; it simply
accepts adequate offers without any such consideration.   Flink does
support a 'host attributes' constraint to filter certain hosts, but that's
not applicable here.

Under the hood, Flink uses a library called Netflix Fenzo to optimize
placement, and a uniqueness constraint could be added by more deeply
leveraging Fenzo's constraint system.   You mentioned that you're trying to
make good use of your GPU resources, which could also be achieved by
treating GPU as a scalar resource (similar to how memory and cores are
treated).   Mesos does support that, but Fenzo may require some
enhancement.   So, these are two potential ways to enhance Flink to support
your scenario.  I'm happy to help; reach out to me.

The obvious, ugly workaround is to configure your TMs to be large enough to
consume the whole host.

Eron





On Thu, Jan 11, 2018 at 7:18 AM, Gary Yao  wrote:

> Hi Dongwon,
>
> I am not familiar with the deployment on DC/OS. However, Eron Wright and
> Jörg
> Schad (cc'd), who have worked on the Mesos integration, might be able to
> help
> you.
>
> Best,
> Gary
>
> On Tue, Jan 9, 2018 at 10:29 AM, Dongwon Kim 
> wrote:
>
>> Hi,
>>
>> I've launched JobManager and TaskManager on DC/OS successfully.
>> Now I have two new issues:
>>
>> 1) All TaskManagers are scheduled on a single node.
>> - Is it intended to maximize data locality and minimize network
>> communication cost?
>> - Is there an option in Flink to adjust the behavior of JobManager when
>> it considers multiple resource offers from different Mesos agents?
>> - I want to schedule TaskManager processes on different GPU servers so
>> that each TaskManger process can use its own GPU cards exclusively.
>> - Below is a part of JobManager log that is occurring while JobManager is
>> negotiating resources with the Mesos master:
>>
>> 2018-01-09 07:34:54,872 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosJobManager  - 
>> JobManager akka.tcp://flink@dnn-g08-233:18026/user/jobmanager was granted 
>> leadership with leader session ID Some(----).
>> 2018-01-09 07:34:55,889 INFO  
>> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connecting 
>> to Mesos...
>> 2018-01-09 07:34:55,962 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Trying to associate with JobManager leader 
>> akka.tcp://flink@dnn-g08-233:18026/user/jobmanager
>> 2018-01-09 07:34:55,977 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Resource Manager associating with leading JobManager 
>> Actor[akka://flink/user/jobmanager#-1481183359] - leader session 
>> ----
>> 2018-01-09 07:34:56,479 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Scheduling Mesos task taskmanager-1 with (10240.0 MB, 8.0 cpus).
>> 2018-01-09 07:34:56,481 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Scheduling Mesos task taskmanager-2 with (10240.0 MB, 8.0 cpus).
>> 2018-01-09 07:34:56,481 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Scheduling Mesos task taskmanager-3 with (10240.0 MB, 8.0 cpus).
>> 2018-01-09 07:34:56,481 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Scheduling Mesos task taskmanager-4 with (10240.0 MB, 8.0 cpus).
>> 2018-01-09 07:34:56,481 INFO  
>> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
>> Scheduling Mesos task taskmanager-5 with (10240.0 MB, 8.0 cpus).
>> 2018-01-09 07:34:56,483 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator- Now 
>> gathering offers for at least 5 task(s).
>> 2018-01-09 07:34:56,484 INFO  
>> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connected to 
>> Mesos as framework ID 59b85b42-a4a2-4632-9578-9e480585ecdc-0004.
>> 2018-01-09 07:34:56,690 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator- Received 
>> offer(s) of 606170.0 MB, 234.2 cpus:
>> 2018-01-09 07:34:56,692 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2174 from 50.1.100.233 of 86.0 MB, 
>> 45.9 cpus for [*]
>> 2018-01-09 07:34:56,692 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2175 from 50.1.100.235 of 123506.0 MB, 
>> 47.3 cpus for [*]
>> 2018-01-09 07:34:56,692 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2176 from 50.1.100.234 of 124530.0 MB, 
>> 46.6 cpus for [*]
>> 2018-01-09 07:34:56,692 INFO  
>> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
>>

Re: Two issues when deploying Flink on DC/OS

2018-01-11 Thread Gary Yao 
Hi Dongwon,

I am not familiar with the deployment on DC/OS. However, Eron Wright and
Jörg
Schad (cc'd), who have worked on the Mesos integration, might be able to
help
you.

Best,
Gary

On Tue, Jan 9, 2018 at 10:29 AM, Dongwon Kim  wrote:

> Hi,
>
> I've launched JobManager and TaskManager on DC/OS successfully.
> Now I have two new issues:
>
> 1) All TaskManagers are scheduled on a single node.
> - Is it intended to maximize data locality and minimize network
> communication cost?
> - Is there an option in Flink to adjust the behavior of JobManager when it
> considers multiple resource offers from different Mesos agents?
> - I want to schedule TaskManager processes on different GPU servers so
> that each TaskManger process can use its own GPU cards exclusively.
> - Below is a part of JobManager log that is occurring while JobManager is
> negotiating resources with the Mesos master:
>
> 2018-01-09 07:34:54,872 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosJobManager  - JobManager 
> akka.tcp://flink@dnn-g08-233:18026/user/jobmanager was granted leadership 
> with leader session ID Some(----).
> 2018-01-09 07:34:55,889 INFO  
> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connecting to 
> Mesos...
> 2018-01-09 07:34:55,962 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Trying to associate with JobManager leader 
> akka.tcp://flink@dnn-g08-233:18026/user/jobmanager
> 2018-01-09 07:34:55,977 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Resource Manager associating with leading JobManager 
> Actor[akka://flink/user/jobmanager#-1481183359] - leader session 
> ----
> 2018-01-09 07:34:56,479 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Scheduling Mesos task taskmanager-1 with (10240.0 MB, 8.0 cpus).
> 2018-01-09 07:34:56,481 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Scheduling Mesos task taskmanager-2 with (10240.0 MB, 8.0 cpus).
> 2018-01-09 07:34:56,481 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Scheduling Mesos task taskmanager-3 with (10240.0 MB, 8.0 cpus).
> 2018-01-09 07:34:56,481 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Scheduling Mesos task taskmanager-4 with (10240.0 MB, 8.0 cpus).
> 2018-01-09 07:34:56,481 INFO  
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager  - 
> Scheduling Mesos task taskmanager-5 with (10240.0 MB, 8.0 cpus).
> 2018-01-09 07:34:56,483 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator- Now gathering 
> offers for at least 5 task(s).
> 2018-01-09 07:34:56,484 INFO  
> org.apache.flink.mesos.scheduler.ConnectionMonitor- Connected to 
> Mesos as framework ID 59b85b42-a4a2-4632-9578-9e480585ecdc-0004.
> 2018-01-09 07:34:56,690 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator- Received 
> offer(s) of 606170.0 MB, 234.2 cpus:
> 2018-01-09 07:34:56,692 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2174 from 50.1.100.233 of 86.0 MB, 
> 45.9 cpus for [*]
> 2018-01-09 07:34:56,692 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2175 from 50.1.100.235 of 123506.0 MB, 
> 47.3 cpus for [*]
> 2018-01-09 07:34:56,692 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2176 from 50.1.100.234 of 124530.0 MB, 
> 46.6 cpus for [*]
> 2018-01-09 07:34:56,692 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2177 from 50.1.100.231 of 123474.0 MB, 
> 47.2 cpus for [*]
> 2018-01-09 07:34:56,693 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 59b85b42-a4a2-4632-9578-9e480585ecdc-O2178 from 50.1.100.232 of 123474.0 MB, 
> 47.2 cpus for [*]
> 2018-01-09 07:34:57,711 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator- Processing 5 
> task(s) against 5 new offer(s) plus outstanding offers.
> 2018-01-09 07:34:57,726 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator- Resources 
> considered: (note: expired offers not deducted from below)
> 2018-01-09 07:34:57,727 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 50.1.100.234 has 124530.0 MB, 46.6 cpus
> 2018-01-09 07:34:57,728 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 50.1.100.235 has 123506.0 MB, 47.3 cpus
> 2018-01-09 07:34:57,728 INFO  
> org.apache.flink.mesos.scheduler.LaunchCoordinator-   
> 50.1.100.232 has 123474.0 MB, 47.2 cpus
> 2018-01-09