Blink是基于Flink 1.5.1做的二次开发,可能存在部分配置和最新版Flink不一致的情况。Sorry 让你踩坑了。
Best,
Kurt
On Fri, Mar 29, 2019 at 5:52 PM 邓成刚【qq】 wrote:
> 终于发现是什么问题了,是由于Blink的 配置与FLINK不同导致:
> Flink 里没有这个配置: taskmanager.cpu.core 默认是 1
>
> 另外:blink 里 taskmanager.heap.mb 与 flink 的 taskmanager.heap.size 不同导致
> taskmanager.heap 配置过小,默认1G
>
> 之前错误的配置:
> # The heap size for the JobManager JVM
>
> jobmanager.heap.size: 20480m
>
>
> # The heap size for the TaskManager JVM
>
> taskmanager.heap.size: 40960m
>
>
> # The number of task slots that each TaskManager offers. Each slot runs
> one parallel pipeline.
>
> taskmanager.numberOfTaskSlots: 24
>
>
>
> 现在配置:
>
>
> # The heap size for the JobManager JVM
>
> jobmanager.heap.mb: 20480
>
>
> # The heap size for the TaskManager JVM
>
> taskmanager.heap.mb: 102400
>
>
> # How many physical cpu cores a task manager will supply for user
>
> taskmanager.cpu.core: 30
>
>
> # The number of task slots that each TaskManager offers. Each slot runs
> one parallel pipeline.
>
> taskmanager.numberOfTaskSlots: 30
>
>
>
> 之前安装了Flink 1.7.2,后又安装 了 Blink,我直接拿的Flink 的配置,没注意看,在此希望我的经历能给大家一些提醒,谢谢。
>
> 在此 感谢龙三 的帮忙,和 谢谢大家的回复。。
>
> 邓成刚【qq】
>
> 发件人: 邓成刚【qq】
> 发送时间: 2019-03-29 17:18
> 收件人: user-zh
> 主题: Re: Re: 请教一下Blink资源分配问题
> 用的是table , LOG里资源申请情况:
>
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> 5ac9229acae1e6ef90563a5a0bf3fe21).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> 2e39ec9bb22b2afe2baa15a87d854796).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id 1a8a3b10d3c235dcb64b4b91e0a22bc8.
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> ce1a10fca1c7a9f6f93dcd248ebce56c).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id b9ea06e5ba61013ead29424b72e8535c.
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id c3742a90378a264341e8d1a573c67535.
>
>
> 发件人: Guowei Ma
> 发送时间: 2019-03-29 17:12
> 收件人: user-zh
> 主题: Re: Re: 请教一下Blink资源分配问题
> 用的什么api,DataStream还是Table?
> 如果是DataStream的话,申请什么样的资源?
> 详细列下。
>
> Best,
> Guowei
>
>
> 邓成刚【qq】 于2019年3月29日周五 下午5:09写道:
>
> > 是的。
> >
> > 发件人: moxian
> > 发送时间: 2019-03-29 17:06
> > 收件人: user-zh
> > 主题: Re: 请教一下Blink资源分配问题
> > standalone 模式?
> >
> > 邓成刚【qq】 于2019年3月29日周五 上午9:59写道:
> >
> > > 请教一下Blink资源分配问题:
> > > blink 任务并行度设置 20 提示0个满足:Batch request 40 slots, but only 0 are
> > fulfilled.
> > > 调整到 3 并行度 提示:Batch request 6 slots, but only 4 are fulfilled.,
> > > 但是我的TASK SLOTS有配 48,没有其它任务,
> > > 按理説没有资源问题啊,集群配置情况:
> > >
> > > 其它的都是默认配置:
> > >
> > > taskmanager.numberOfTaskSlots: 24
> > >
> > > jobmanager.heap.size: 20480m
> > >
> > > # The heap size for the TaskManager JVM
> > >
> > > taskmanager.heap.size: 40960m
> > >
> > >
> > > 服务器 2 台:每台 48核,256G
> > >
> > >
> > >