Re: Container JMX port setting / discovery for Flink on YARN

[Yury Ruchin]

Thanks Robert, it works like a charm.

2016-11-25 12:55 GMT+03:00 Robert Metzger :

> Hi Yury,
>
> Flink is using its own JMX server instance (not the JVM's one). Therefore,
> you can configure the server yourself.
> Check out this documentation page: https://ci.apache.org/
> projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter
>
> metrics.reporter.my_jmx_reporter.class: 
> org.apache.flink.metrics.jmx.JMXReporter
> metrics.reporter.my_jmx_reporter.port: 9020-9040
>
>
> Flink will print the port it is using in the end into the log of each
> TaskManager.
>
>
> On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin 
> wrote:
>
>> Thanks Stefan! I think this would help if I had just one container per
>> node. But that's not my case - there are multiple TaskManagers running on
>> the same node, so setting the same value will likely result in port
>> conflict.
>>
>> 2016-11-25 12:28 GMT+03:00 Stefan Richter :
>>
>>> Hi,
>>>
>>> can you try adding the following to your flink.yaml?
>>>
>>> env.java.opts: -Dcom.sun.management.jmxremote
>>> -Dcom.sun.management.jmxremote.port=
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Dcom.sun.management.jmxremote.ssl=false
>>>
>>> Best,
>>> Stefan
>>>
>>>
>>> > Am 24.11.2016 um 16:47 schrieb Yury Ruchin :
>>> >
>>> > Hi,
>>> >
>>> > I want to enable JMX for my Flink streaming app running in YARN
>>> session. How can I specify which ports containers will listen to? If I
>>> cannot control it (e. g. they will be chosen randomly) - how can I detect
>>> which ports were picked by containers: inspecting logs, looking at the Web
>>> UI etc.?
>>> >
>>> > Example: in Apache Storm it is easy to derive JMX port numbers from
>>> worker port (slot) numbers and pass it as a JVM argument to the worker
>>> start command, so that every slot is always associated with a stable and
>>> well-known JMX port.
>>> >
>>> > Any clue is appreciated. Thanks!
>>>
>>>
>>
>

Re: Container JMX port setting / discovery for Flink on YARN

[Robert Metzger]

Hi Yury,

Flink is using its own JMX server instance (not the JVM's one). Therefore,
you can configure the server yourself.
Check out this documentation page:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter

metrics.reporter.my_jmx_reporter.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.my_jmx_reporter.port: 9020-9040


Flink will print the port it is using in the end into the log of each
TaskManager.


On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin  wrote:

> Thanks Stefan! I think this would help if I had just one container per
> node. But that's not my case - there are multiple TaskManagers running on
> the same node, so setting the same value will likely result in port
> conflict.
>
> 2016-11-25 12:28 GMT+03:00 Stefan Richter :
>
>> Hi,
>>
>> can you try adding the following to your flink.yaml?
>>
>> env.java.opts: -Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.port=
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false
>>
>> Best,
>> Stefan
>>
>>
>> > Am 24.11.2016 um 16:47 schrieb Yury Ruchin :
>> >
>> > Hi,
>> >
>> > I want to enable JMX for my Flink streaming app running in YARN
>> session. How can I specify which ports containers will listen to? If I
>> cannot control it (e. g. they will be chosen randomly) - how can I detect
>> which ports were picked by containers: inspecting logs, looking at the Web
>> UI etc.?
>> >
>> > Example: in Apache Storm it is easy to derive JMX port numbers from
>> worker port (slot) numbers and pass it as a JVM argument to the worker
>> start command, so that every slot is always associated with a stable and
>> well-known JMX port.
>> >
>> > Any clue is appreciated. Thanks!
>>
>>
>

Re: Container JMX port setting / discovery for Flink on YARN

[Yury Ruchin]

Thanks Stefan! I think this would help if I had just one container per
node. But that's not my case - there are multiple TaskManagers running on
the same node, so setting the same value will likely result in port
conflict.

2016-11-25 12:28 GMT+03:00 Stefan Richter :

> Hi,
>
> can you try adding the following to your flink.yaml?
>
> env.java.opts: -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
>
> Best,
> Stefan
>
>
> > Am 24.11.2016 um 16:47 schrieb Yury Ruchin :
> >
> > Hi,
> >
> > I want to enable JMX for my Flink streaming app running in YARN session.
> How can I specify which ports containers will listen to? If I cannot
> control it (e. g. they will be chosen randomly) - how can I detect which
> ports were picked by containers: inspecting logs, looking at the Web UI
> etc.?
> >
> > Example: in Apache Storm it is easy to derive JMX port numbers from
> worker port (slot) numbers and pass it as a JVM argument to the worker
> start command, so that every slot is always associated with a stable and
> well-known JMX port.
> >
> > Any clue is appreciated. Thanks!
>
>

Re: Container JMX port setting / discovery for Flink on YARN

[Stefan Richter]

Hi,

can you try adding the following to your flink.yaml?

env.java.opts: -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port= 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false

Best,
Stefan


> Am 24.11.2016 um 16:47 schrieb Yury Ruchin :
> 
> Hi,
> 
> I want to enable JMX for my Flink streaming app running in YARN session. How 
> can I specify which ports containers will listen to? If I cannot control it 
> (e. g. they will be chosen randomly) - how can I detect which ports were 
> picked by containers: inspecting logs, looking at the Web UI etc.?
> 
> Example: in Apache Storm it is easy to derive JMX port numbers from worker 
> port (slot) numbers and pass it as a JVM argument to the worker start 
> command, so that every slot is always associated with a stable and well-known 
> JMX port.
> 
> Any clue is appreciated. Thanks!