Re: taskmanager.cpu.cores 1.7976931348623157E308

[Rex Fenley]

Thanks for all this feedback, this is really helpful!

On Sun, Dec 6, 2020 at 7:23 PM Xintong Song  wrote:

> FYI, I've opened FLINK-20503 for this.
> https://issues.apache.org/jira/browse/FLINK-20503
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Dec 7, 2020 at 11:10 AM Xintong Song 
> wrote:
>
>> I forgot to mention that it is designed that task managers always have
>> `Double#MAX_VALUE` cpu cores in local execution.
>>
>>
>> And I think Yangze is right. The log "The configuration option
>> taskmanager.cpu.cores required for local execution is not set, setting
>> it to" can be misleading for users. Will fire an issue on that.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>>
>> On Mon, Dec 7, 2020 at 11:03 AM Xintong Song 
>> wrote:
>>
>>> Hi Rex,
>>>
>>> We're running this in a local environment so that may be contributing to
 what we're seeing.

>>> Just to double check on this. By `local environment`, you mean running
>>> flink without setting up a standalone cluster or submitting it to a
>>> K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t
>>> local`, or running your own application that calls
>>> `ExecutionEnvironment#execute`).
>>> If yes, then this is kind of expected.
>>>
>>> A couple of things that might help you understand this.
>>>
>>>- Running on a local environment means setting up the Flink cluster
>>>within the current process (the IDE process if executed from an IDE, the
>>>flink client process if using `flink run -t local`, or your own 
>>> application
>>>process). That also means most of the resource configurations cannot take
>>>effect, because the resources of the JVM are already determined. Please
>>>refer to the memory configuration documents for options that still take
>>>effect in local execution. [1][2]
>>>- David is correct that `taskmanager.cpu.cores` is only intended for
>>>internal usages. I assume you learnt about this configuration by reading
>>>the source codes? If true, please also be aware that the JavaDoc
>>>of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", 
>>> and
>>>it is also annotated with `ExcludeFromDocumentation` so that users do not
>>>learn this option from the documents.
>>>- Flink does not really control how many cpu cores it uses. However,
>>>when running on an external resource management system (K8s, Yarn, 
>>> Mesos),
>>>it requires a certain amount of cpu resources for its containers/pods, 
>>> and
>>>allows the external system to control its cpu usage. You can use the
>>>following configuration options to control how many cpu cores are 
>>> requested
>>>in such cases.
>>>   - kubernetes.jobmanager.cpu
>>>   - kubernetes.taskmanager.cpu
>>>   - yarn.appmaster.vcores
>>>   - yarn.containers.vcores
>>>   - mesos.resourcemanager.tasks.cpus
>>>
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution
>>> [2]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution
>>>
>>>
>>> On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo  wrote:
>>>
 Hi, Rex,

 Can you share more logs for it. Did you see something like "The
 configuration option taskmanager.cpu.cores required for local
 execution is not set, setting it to" in your logs?

 Best,
 Yangze Guo

 Best,
 Yangze Guo


 On Sat, Dec 5, 2020 at 6:53 PM David Anderson 
 wrote:
 >
 > taskmanager.cpu.cores is intended for internal use only -- you aren't
 meant to set this option. What happens if you leave it alone?
 >
 > Regards,
 > David
 >
 >
 > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
 >>
 >> We're running this in a local environment so that may be
 contributing to what we're seeing.
 >>
 >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley 
 wrote:
 >>>
 >>> Hello,
 >>>
 >>> I'm tuning flink for parallelism right now and when I look at the
 JobManager I see
 >>> taskmanager.cpu.cores1.7976931348623157E308
 >>> Which looks like the maximum double number.
 >>>
 >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper
 threading. We have 37 operators so we rounded up and set 40 task slots.
 >>>
 >>> Here is our configuration
 >>>
 >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552
 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log
 -Dtaskmanager.memory.framework.off-heap.size=134217728b
 -Dtaskmanager.memory.network.max=1073741824b
 -Dtaskmanager.memory.network.min=1073741824b
 -Dtaskmanager.memory.framework.heap.size=134217728b
 -Dtaskmanager.memory.managed.size=6335076856b
 -Dtaskmanager.memory.task.heap.size=8160437768b
 -Dtaskmanager.memory.task.off-heap.size=0b

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Xintong Song]

FYI, I've opened FLINK-20503 for this.
https://issues.apache.org/jira/browse/FLINK-20503

Thank you~

Xintong Song



On Mon, Dec 7, 2020 at 11:10 AM Xintong Song  wrote:

> I forgot to mention that it is designed that task managers always have
> `Double#MAX_VALUE` cpu cores in local execution.
>
>
> And I think Yangze is right. The log "The configuration option
> taskmanager.cpu.cores required for local execution is not set, setting it
> to" can be misleading for users. Will fire an issue on that.
>
>
> Thank you~
>
> Xintong Song
>
>
>
>
> On Mon, Dec 7, 2020 at 11:03 AM Xintong Song 
> wrote:
>
>> Hi Rex,
>>
>> We're running this in a local environment so that may be contributing to
>>> what we're seeing.
>>>
>> Just to double check on this. By `local environment`, you mean running
>> flink without setting up a standalone cluster or submitting it to a
>> K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t
>> local`, or running your own application that calls
>> `ExecutionEnvironment#execute`).
>> If yes, then this is kind of expected.
>>
>> A couple of things that might help you understand this.
>>
>>- Running on a local environment means setting up the Flink cluster
>>within the current process (the IDE process if executed from an IDE, the
>>flink client process if using `flink run -t local`, or your own 
>> application
>>process). That also means most of the resource configurations cannot take
>>effect, because the resources of the JVM are already determined. Please
>>refer to the memory configuration documents for options that still take
>>effect in local execution. [1][2]
>>- David is correct that `taskmanager.cpu.cores` is only intended for
>>internal usages. I assume you learnt about this configuration by reading
>>the source codes? If true, please also be aware that the JavaDoc
>>of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", and
>>it is also annotated with `ExcludeFromDocumentation` so that users do not
>>learn this option from the documents.
>>- Flink does not really control how many cpu cores it uses. However,
>>when running on an external resource management system (K8s, Yarn, Mesos),
>>it requires a certain amount of cpu resources for its containers/pods, and
>>allows the external system to control its cpu usage. You can use the
>>following configuration options to control how many cpu cores are 
>> requested
>>in such cases.
>>   - kubernetes.jobmanager.cpu
>>   - kubernetes.taskmanager.cpu
>>   - yarn.appmaster.vcores
>>   - yarn.containers.vcores
>>   - mesos.resourcemanager.tasks.cpus
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution
>>
>>
>> On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo  wrote:
>>
>>> Hi, Rex,
>>>
>>> Can you share more logs for it. Did you see something like "The
>>> configuration option taskmanager.cpu.cores required for local
>>> execution is not set, setting it to" in your logs?
>>>
>>> Best,
>>> Yangze Guo
>>>
>>> Best,
>>> Yangze Guo
>>>
>>>
>>> On Sat, Dec 5, 2020 at 6:53 PM David Anderson 
>>> wrote:
>>> >
>>> > taskmanager.cpu.cores is intended for internal use only -- you aren't
>>> meant to set this option. What happens if you leave it alone?
>>> >
>>> > Regards,
>>> > David
>>> >
>>> >
>>> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
>>> >>
>>> >> We're running this in a local environment so that may be contributing
>>> to what we're seeing.
>>> >>
>>> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> I'm tuning flink for parallelism right now and when I look at the
>>> JobManager I see
>>> >>> taskmanager.cpu.cores1.7976931348623157E308
>>> >>> Which looks like the maximum double number.
>>> >>>
>>> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper
>>> threading. We have 37 operators so we rounded up and set 40 task slots.
>>> >>>
>>> >>> Here is our configuration
>>> >>>
>>> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552
>>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log
>>> -Dtaskmanager.memory.framework.off-heap.size=134217728b
>>> -Dtaskmanager.memory.network.max=1073741824b
>>> -Dtaskmanager.memory.network.min=1073741824b
>>> -Dtaskmanager.memory.framework.heap.size=134217728b
>>> -Dtaskmanager.memory.managed.size=6335076856b
>>> -Dtaskmanager.memory.task.heap.size=8160437768b
>>> -Dtaskmanager.memory.task.off-heap.size=0b
>>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
>>> >>>
>>> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up
>>> with that very odd value for cpu cores.
>>> >>>
>>> >>> How do we correctly adjust this?
>>> >>>
>>> >>> Thanks!
>>> 

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Xintong Song]

I forgot to mention that it is designed that task managers always have
`Double#MAX_VALUE` cpu cores in local execution.


And I think Yangze is right. The log "The configuration option
taskmanager.cpu.cores required for local execution is not set, setting it
to" can be misleading for users. Will fire an issue on that.


Thank you~

Xintong Song




On Mon, Dec 7, 2020 at 11:03 AM Xintong Song  wrote:

> Hi Rex,
>
> We're running this in a local environment so that may be contributing to
>> what we're seeing.
>>
> Just to double check on this. By `local environment`, you mean running
> flink without setting up a standalone cluster or submitting it to a
> K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t
> local`, or running your own application that calls
> `ExecutionEnvironment#execute`).
> If yes, then this is kind of expected.
>
> A couple of things that might help you understand this.
>
>- Running on a local environment means setting up the Flink cluster
>within the current process (the IDE process if executed from an IDE, the
>flink client process if using `flink run -t local`, or your own application
>process). That also means most of the resource configurations cannot take
>effect, because the resources of the JVM are already determined. Please
>refer to the memory configuration documents for options that still take
>effect in local execution. [1][2]
>- David is correct that `taskmanager.cpu.cores` is only intended for
>internal usages. I assume you learnt about this configuration by reading
>the source codes? If true, please also be aware that the JavaDoc
>of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", and
>it is also annotated with `ExcludeFromDocumentation` so that users do not
>learn this option from the documents.
>- Flink does not really control how many cpu cores it uses. However,
>when running on an external resource management system (K8s, Yarn, Mesos),
>it requires a certain amount of cpu resources for its containers/pods, and
>allows the external system to control its cpu usage. You can use the
>following configuration options to control how many cpu cores are requested
>in such cases.
>   - kubernetes.jobmanager.cpu
>   - kubernetes.taskmanager.cpu
>   - yarn.appmaster.vcores
>   - yarn.containers.vcores
>   - mesos.resourcemanager.tasks.cpus
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution
>
>
> On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo  wrote:
>
>> Hi, Rex,
>>
>> Can you share more logs for it. Did you see something like "The
>> configuration option taskmanager.cpu.cores required for local
>> execution is not set, setting it to" in your logs?
>>
>> Best,
>> Yangze Guo
>>
>> Best,
>> Yangze Guo
>>
>>
>> On Sat, Dec 5, 2020 at 6:53 PM David Anderson 
>> wrote:
>> >
>> > taskmanager.cpu.cores is intended for internal use only -- you aren't
>> meant to set this option. What happens if you leave it alone?
>> >
>> > Regards,
>> > David
>> >
>> >
>> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
>> >>
>> >> We're running this in a local environment so that may be contributing
>> to what we're seeing.
>> >>
>> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> I'm tuning flink for parallelism right now and when I look at the
>> JobManager I see
>> >>> taskmanager.cpu.cores1.7976931348623157E308
>> >>> Which looks like the maximum double number.
>> >>>
>> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper
>> threading. We have 37 operators so we rounded up and set 40 task slots.
>> >>>
>> >>> Here is our configuration
>> >>>
>> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552
>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log
>> -Dtaskmanager.memory.framework.off-heap.size=134217728b
>> -Dtaskmanager.memory.network.max=1073741824b
>> -Dtaskmanager.memory.network.min=1073741824b
>> -Dtaskmanager.memory.framework.heap.size=134217728b
>> -Dtaskmanager.memory.managed.size=6335076856b
>> -Dtaskmanager.memory.task.heap.size=8160437768b
>> -Dtaskmanager.memory.task.off-heap.size=0b
>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
>> >>>
>> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up
>> with that very odd value for cpu cores.
>> >>>
>> >>> How do we correctly adjust this?
>> >>>
>> >>> Thanks!
>> >>> --
>> >>>
>> >>> Rex Fenley  |  Software Engineer - Mobile and Backend
>> >>>
>> >>>
>> >>> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Rex Fenley  |  Software Engineer - Mobile and Backend
>> >>
>> >>
>> >> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
>>
>

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Xintong Song]

Hi Rex,

We're running this in a local environment so that may be contributing to
> what we're seeing.
>
Just to double check on this. By `local environment`, you mean running
flink without setting up a standalone cluster or submitting it to a
K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t
local`, or running your own application that calls
`ExecutionEnvironment#execute`).
If yes, then this is kind of expected.

A couple of things that might help you understand this.

   - Running on a local environment means setting up the Flink cluster
   within the current process (the IDE process if executed from an IDE, the
   flink client process if using `flink run -t local`, or your own application
   process). That also means most of the resource configurations cannot take
   effect, because the resources of the JVM are already determined. Please
   refer to the memory configuration documents for options that still take
   effect in local execution. [1][2]
   - David is correct that `taskmanager.cpu.cores` is only intended for
   internal usages. I assume you learnt about this configuration by reading
   the source codes? If true, please also be aware that the JavaDoc
   of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", and
   it is also annotated with `ExcludeFromDocumentation` so that users do not
   learn this option from the documents.
   - Flink does not really control how many cpu cores it uses. However,
   when running on an external resource management system (K8s, Yarn, Mesos),
   it requires a certain amount of cpu resources for its containers/pods, and
   allows the external system to control its cpu usage. You can use the
   following configuration options to control how many cpu cores are requested
   in such cases.
  - kubernetes.jobmanager.cpu
  - kubernetes.taskmanager.cpu
  - yarn.appmaster.vcores
  - yarn.containers.vcores
  - mesos.resourcemanager.tasks.cpus


Thank you~

Xintong Song


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution


On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo  wrote:

> Hi, Rex,
>
> Can you share more logs for it. Did you see something like "The
> configuration option taskmanager.cpu.cores required for local
> execution is not set, setting it to" in your logs?
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Sat, Dec 5, 2020 at 6:53 PM David Anderson 
> wrote:
> >
> > taskmanager.cpu.cores is intended for internal use only -- you aren't
> meant to set this option. What happens if you leave it alone?
> >
> > Regards,
> > David
> >
> >
> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
> >>
> >> We're running this in a local environment so that may be contributing
> to what we're seeing.
> >>
> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm tuning flink for parallelism right now and when I look at the
> JobManager I see
> >>> taskmanager.cpu.cores1.7976931348623157E308
> >>> Which looks like the maximum double number.
> >>>
> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper
> threading. We have 37 operators so we rounded up and set 40 task slots.
> >>>
> >>> Here is our configuration
> >>>
> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552
> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log
> -Dtaskmanager.memory.framework.off-heap.size=134217728b
> -Dtaskmanager.memory.network.max=1073741824b
> -Dtaskmanager.memory.network.min=1073741824b
> -Dtaskmanager.memory.framework.heap.size=134217728b
> -Dtaskmanager.memory.managed.size=6335076856b
> -Dtaskmanager.memory.task.heap.size=8160437768b
> -Dtaskmanager.memory.task.off-heap.size=0b
> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
> >>>
> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with
> that very odd value for cpu cores.
> >>>
> >>> How do we correctly adjust this?
> >>>
> >>> Thanks!
> >>> --
> >>>
> >>> Rex Fenley  |  Software Engineer - Mobile and Backend
> >>>
> >>>
> >>> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
> >>
> >>
> >>
> >> --
> >>
> >> Rex Fenley  |  Software Engineer - Mobile and Backend
> >>
> >>
> >> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
>

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Yangze Guo]

My gut feeling is your "vmArgs" does not take effect.

Best,
Yangze Guo

On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo  wrote:
>
> Hi, Rex,
>
> Can you share more logs for it. Did you see something like "The
> configuration option taskmanager.cpu.cores required for local
> execution is not set, setting it to" in your logs?
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Sat, Dec 5, 2020 at 6:53 PM David Anderson  wrote:
> >
> > taskmanager.cpu.cores is intended for internal use only -- you aren't meant 
> > to set this option. What happens if you leave it alone?
> >
> > Regards,
> > David
> >
> >
> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
> >>
> >> We're running this in a local environment so that may be contributing to 
> >> what we're seeing.
> >>
> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm tuning flink for parallelism right now and when I look at the 
> >>> JobManager I see
> >>> taskmanager.cpu.cores1.7976931348623157E308
> >>> Which looks like the maximum double number.
> >>>
> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper threading. 
> >>> We have 37 operators so we rounded up and set 40 task slots.
> >>>
> >>> Here is our configuration
> >>>
> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552 
> >>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log 
> >>> -Dtaskmanager.memory.framework.off-heap.size=134217728b 
> >>> -Dtaskmanager.memory.network.max=1073741824b 
> >>> -Dtaskmanager.memory.network.min=1073741824b 
> >>> -Dtaskmanager.memory.framework.heap.size=134217728b 
> >>> -Dtaskmanager.memory.managed.size=6335076856b 
> >>> -Dtaskmanager.memory.task.heap.size=8160437768b 
> >>> -Dtaskmanager.memory.task.off-heap.size=0b 
> >>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
> >>>
> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with 
> >>> that very odd value for cpu cores.
> >>>
> >>> How do we correctly adjust this?
> >>>
> >>> Thanks!
> >>> --
> >>>
> >>> Rex Fenley  |  Software Engineer - Mobile and Backend
> >>>
> >>>
> >>> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
> >>
> >>
> >>
> >> --
> >>
> >> Rex Fenley  |  Software Engineer - Mobile and Backend
> >>
> >>
> >> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Yangze Guo]

Hi, Rex,

Can you share more logs for it. Did you see something like "The
configuration option taskmanager.cpu.cores required for local
execution is not set, setting it to" in your logs?

Best,
Yangze Guo

Best,
Yangze Guo


On Sat, Dec 5, 2020 at 6:53 PM David Anderson  wrote:
>
> taskmanager.cpu.cores is intended for internal use only -- you aren't meant 
> to set this option. What happens if you leave it alone?
>
> Regards,
> David
>
>
> On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:
>>
>> We're running this in a local environment so that may be contributing to 
>> what we're seeing.
>>
>> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
>>>
>>> Hello,
>>>
>>> I'm tuning flink for parallelism right now and when I look at the 
>>> JobManager I see
>>> taskmanager.cpu.cores1.7976931348623157E308
>>> Which looks like the maximum double number.
>>>
>>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper threading. We 
>>> have 37 operators so we rounded up and set 40 task slots.
>>>
>>> Here is our configuration
>>>
>>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552 
>>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log 
>>> -Dtaskmanager.memory.framework.off-heap.size=134217728b 
>>> -Dtaskmanager.memory.network.max=1073741824b 
>>> -Dtaskmanager.memory.network.min=1073741824b 
>>> -Dtaskmanager.memory.framework.heap.size=134217728b 
>>> -Dtaskmanager.memory.managed.size=6335076856b 
>>> -Dtaskmanager.memory.task.heap.size=8160437768b 
>>> -Dtaskmanager.memory.task.off-heap.size=0b 
>>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
>>>
>>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with that 
>>> very odd value for cpu cores.
>>>
>>> How do we correctly adjust this?
>>>
>>> Thanks!
>>> --
>>>
>>> Rex Fenley  |  Software Engineer - Mobile and Backend
>>>
>>>
>>> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US
>>
>>
>>
>> --
>>
>> Rex Fenley  |  Software Engineer - Mobile and Backend
>>
>>
>> Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

Re: taskmanager.cpu.cores 1.7976931348623157E308

[David Anderson]

taskmanager.cpu.cores is intended for internal use only -- you aren't meant
to set this option. What happens if you leave it alone?

Regards,
David


On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley  wrote:

> We're running this in a local environment so that may be contributing to
> what we're seeing.
>
> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:
>
>> Hello,
>>
>> I'm tuning flink for parallelism right now and when I look at the
>> JobManager I see
>> *taskmanager.cpu.cores* 1.7976931348623157E308
>> Which looks like the maximum double number.
>>
>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper threading.
>> We have 37 operators so we rounded up and set 40 task slots.
>>
>> Here is our configuration
>>
>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552 
>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log 
>> -Dtaskmanager.memory.framework.off-heap.size=134217728b 
>> -Dtaskmanager.memory.network.max=1073741824b 
>> -Dtaskmanager.memory.network.min=1073741824b 
>> -Dtaskmanager.memory.framework.heap.size=134217728b 
>> -Dtaskmanager.memory.managed.size=6335076856b 
>> -Dtaskmanager.memory.task.heap.size=8160437768b 
>> -Dtaskmanager.memory.task.off-heap.size=0b 
>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
>>
>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with
>> that very odd value for cpu cores.
>>
>> How do we correctly adjust this?
>>
>> Thanks!
>> --
>>
>> Rex Fenley  |  Software Engineer - Mobile and Backend
>>
>>
>> Remind.com  |  BLOG 
>>  |  FOLLOW US   |  LIKE US
>> 
>>
>
>
> --
>
> Rex Fenley  |  Software Engineer - Mobile and Backend
>
>
> Remind.com  |  BLOG   |
>  FOLLOW US   |  LIKE US
> 
>

Re: taskmanager.cpu.cores 1.7976931348623157E308

[Rex Fenley]

We're running this in a local environment so that may be contributing to
what we're seeing.

On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley  wrote:

> Hello,
>
> I'm tuning flink for parallelism right now and when I look at the
> JobManager I see
> *taskmanager.cpu.cores* 1.7976931348623157E308
> Which looks like the maximum double number.
>
> We have 8 cpu cores, so we figured we'd bump to 16 for hyper threading. We
> have 37 operators so we rounded up and set 40 task slots.
>
> Here is our configuration
>
> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552 
> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log 
> -Dtaskmanager.memory.framework.off-heap.size=134217728b 
> -Dtaskmanager.memory.network.max=1073741824b 
> -Dtaskmanager.memory.network.min=1073741824b 
> -Dtaskmanager.memory.framework.heap.size=134217728b 
> -Dtaskmanager.memory.managed.size=6335076856b 
> -Dtaskmanager.memory.task.heap.size=8160437768b 
> -Dtaskmanager.memory.task.off-heap.size=0b -Dtaskmanager.numberOfTaskSlots=40 
> -Dtaskmanager.cpu.cores=16.0"
>
> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with
> that very odd value for cpu cores.
>
> How do we correctly adjust this?
>
> Thanks!
> --
>
> Rex Fenley  |  Software Engineer - Mobile and Backend
>
>
> Remind.com  |  BLOG   |
>  FOLLOW US   |  LIKE US
> 
>


-- 

Rex Fenley  |  Software Engineer - Mobile and Backend


Remind.com  |  BLOG 
 |  FOLLOW
US   |  LIKE US