Thanks for all this feedback, this is really helpful! On Sun, Dec 6, 2020 at 7:23 PM Xintong Song <tonysong...@gmail.com> wrote:
> FYI, I've opened FLINK-20503 for this. > https://issues.apache.org/jira/browse/FLINK-20503 > > Thank you~ > > Xintong Song > > > > On Mon, Dec 7, 2020 at 11:10 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> I forgot to mention that it is designed that task managers always have >> `Double#MAX_VALUE` cpu cores in local execution. >> >> >> And I think Yangze is right. The log "The configuration option >> taskmanager.cpu.cores required for local execution is not set, setting >> it to" can be misleading for users. Will fire an issue on that. >> >> >> Thank you~ >> >> Xintong Song >> >> >> >> >> On Mon, Dec 7, 2020 at 11:03 AM Xintong Song <tonysong...@gmail.com> >> wrote: >> >>> Hi Rex, >>> >>> We're running this in a local environment so that may be contributing to >>>> what we're seeing. >>>> >>> Just to double check on this. By `local environment`, you mean running >>> flink without setting up a standalone cluster or submitting it to a >>> K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t >>> local`, or running your own application that calls >>> `ExecutionEnvironment#execute`). >>> If yes, then this is kind of expected. >>> >>> A couple of things that might help you understand this. >>> >>> - Running on a local environment means setting up the Flink cluster >>> within the current process (the IDE process if executed from an IDE, the >>> flink client process if using `flink run -t local`, or your own >>> application >>> process). That also means most of the resource configurations cannot take >>> effect, because the resources of the JVM are already determined. Please >>> refer to the memory configuration documents for options that still take >>> effect in local execution. [1][2] >>> - David is correct that `taskmanager.cpu.cores` is only intended for >>> internal usages. I assume you learnt about this configuration by reading >>> the source codes? If true, please also be aware that the JavaDoc >>> of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", >>> and >>> it is also annotated with `ExcludeFromDocumentation` so that users do not >>> learn this option from the documents. >>> - Flink does not really control how many cpu cores it uses. However, >>> when running on an external resource management system (K8s, Yarn, >>> Mesos), >>> it requires a certain amount of cpu resources for its containers/pods, >>> and >>> allows the external system to control its cpu usage. You can use the >>> following configuration options to control how many cpu cores are >>> requested >>> in such cases. >>> - kubernetes.jobmanager.cpu >>> - kubernetes.taskmanager.cpu >>> - yarn.appmaster.vcores >>> - yarn.containers.vcores >>> - mesos.resourcemanager.tasks.cpus >>> >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution >>> [2] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution >>> >>> >>> On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo <karma...@gmail.com> wrote: >>> >>>> Hi, Rex, >>>> >>>> Can you share more logs for it. Did you see something like "The >>>> configuration option taskmanager.cpu.cores required for local >>>> execution is not set, setting it to" in your logs? >>>> >>>> Best, >>>> Yangze Guo >>>> >>>> Best, >>>> Yangze Guo >>>> >>>> >>>> On Sat, Dec 5, 2020 at 6:53 PM David Anderson <dander...@apache.org> >>>> wrote: >>>> > >>>> > taskmanager.cpu.cores is intended for internal use only -- you aren't >>>> meant to set this option. What happens if you leave it alone? >>>> > >>>> > Regards, >>>> > David >>>> > >>>> > >>>> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley <r...@remind101.com> wrote: >>>> >> >>>> >> We're running this in a local environment so that may be >>>> contributing to what we're seeing. >>>> >> >>>> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley <r...@remind101.com> >>>> wrote: >>>> >>> >>>> >>> Hello, >>>> >>> >>>> >>> I'm tuning flink for parallelism right now and when I look at the >>>> JobManager I see >>>> >>> taskmanager.cpu.cores1.7976931348623157E308 >>>> >>> Which looks like the maximum double number. >>>> >>> >>>> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper >>>> threading. We have 37 operators so we rounded up and set 40 task slots. >>>> >>> >>>> >>> Here is our configuration >>>> >>> >>>> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552 >>>> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log >>>> -Dtaskmanager.memory.framework.off-heap.size=134217728b >>>> -Dtaskmanager.memory.network.max=1073741824b >>>> -Dtaskmanager.memory.network.min=1073741824b >>>> -Dtaskmanager.memory.framework.heap.size=134217728b >>>> -Dtaskmanager.memory.managed.size=6335076856b >>>> -Dtaskmanager.memory.task.heap.size=8160437768b >>>> -Dtaskmanager.memory.task.off-heap.size=0b >>>> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0" >>>> >>> >>>> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up >>>> with that very odd value for cpu cores. >>>> >>> >>>> >>> How do we correctly adjust this? >>>> >>> >>>> >>> Thanks! >>>> >>> -- >>>> >>> >>>> >>> Rex Fenley | Software Engineer - Mobile and Backend >>>> >>> >>>> >>> >>>> >>> Remind.com | BLOG | FOLLOW US | LIKE US >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> >>>> >> Rex Fenley | Software Engineer - Mobile and Backend >>>> >> >>>> >> >>>> >> Remind.com | BLOG | FOLLOW US | LIKE US >>>> >>> -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | FOLLOW US <https://twitter.com/remindhq> | LIKE US <https://www.facebook.com/remindhq>
