Re: 结邮 Re: Re: 请教一下Blink资源分配问题

2019-03-29 文章 Kurt Young
Blink是基于Flink 1.5.1做的二次开发,可能存在部分配置和最新版Flink不一致的情况。Sorry 让你踩坑了。

Best,
Kurt


On Fri, Mar 29, 2019 at 5:52 PM 邓成刚【qq】  wrote:

> 终于发现是什么问题了,是由于Blink的 配置与FLINK不同导致:
> Flink 里没有这个配置:  taskmanager.cpu.core  默认是   1
>
> 另外:blink 里 taskmanager.heap.mb   与 flink 的 taskmanager.heap.size  不同导致
>   taskmanager.heap 配置过小,默认1G
>
> 之前错误的配置:
> # The heap size for the JobManager JVM
>
> jobmanager.heap.size: 20480m
>
>
> # The heap size for the TaskManager JVM
>
> taskmanager.heap.size: 40960m
>
>
> # The number of task slots that each TaskManager offers. Each slot runs
> one parallel pipeline.
>
> taskmanager.numberOfTaskSlots: 24
>
>
>
> 现在配置:
>
>
> # The heap size for the JobManager JVM
>
> jobmanager.heap.mb: 20480
>
>
> # The heap size for the TaskManager JVM
>
> taskmanager.heap.mb: 102400
>
>
> # How many physical cpu cores a task manager will supply for user
>
> taskmanager.cpu.core: 30
>
>
> # The number of task slots that each TaskManager offers. Each slot runs
> one parallel pipeline.
>
> taskmanager.numberOfTaskSlots: 30
>
>
>
> 之前安装了Flink 1.7.2,后又安装 了 Blink,我直接拿的Flink 的配置,没注意看,在此希望我的经历能给大家一些提醒,谢谢。
>
> 在此 感谢龙三 的帮忙,和  谢谢大家的回复。。
>
> 邓成刚【qq】
>
> 发件人: 邓成刚【qq】
> 发送时间: 2019-03-29 17:18
> 收件人: user-zh
> 主题: Re: Re: 请教一下Blink资源分配问题
> 用的是table , LOG里资源申请情况:
>
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> 5ac9229acae1e6ef90563a5a0bf3fe21).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> 2e39ec9bb22b2afe2baa15a87d854796).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id 1a8a3b10d3c235dcb64b4b91e0a22bc8.
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request =
> ce1a10fca1c7a9f6f93dcd248ebce56c).
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id b9ea06e5ba61013ead29424b72e8535c.
> ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0,
> nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0,
> FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3
> with allocation id c3742a90378a264341e8d1a573c67535.
>
>
> 发件人: Guowei Ma
> 发送时间: 2019-03-29 17:12
> 收件人: user-zh
> 主题: Re: Re: 请教一下Blink资源分配问题
> 用的什么api,DataStream还是Table?
> 如果是DataStream的话,申请什么样的资源?
> 详细列下。
>
> Best,
> Guowei
>
>
> 邓成刚【qq】  于2019年3月29日周五 下午5:09写道:
>
> > 是的。
> >
> > 发件人: moxian
> > 发送时间: 2019-03-29 17:06
> > 收件人: user-zh
> > 主题: Re: 请教一下Blink资源分配问题
> > standalone 模式?
> >
> > 邓成刚【qq】  于2019年3月29日周五 上午9:59写道:
> >
> > > 请教一下Blink资源分配问题:
> > > blink 任务并行度设置 20  提示0个满足:Batch request 40 slots, but only 0 are
> > fulfilled.
> > > 调整到 3 并行度 提示:Batch request 6 slots, but only 4 are fulfilled.,
> > > 但是我的TASK SLOTS有配 48,没有其它任务,
> > > 按理説没有资源问题啊,集群配置情况:
> > >
> > > 其它的都是默认配置:
> > >
> > > taskmanager.numberOfTaskSlots: 24
> > >
> > > jobmanager.heap.size: 20480m
> > >
> > > # The heap size for the TaskManager JVM
> > >
> > > taskmanager.heap.size: 40960m
> > >
> > >
> > > 服务器 2 台:每台 48核,256G
> > >
> > >
> > >


结邮 Re: Re: 请教一下Blink资源分配问题

2019-03-29 文章 邓成刚【qq】
终于发现是什么问题了,是由于Blink的 配置与FLINK不同导致:
Flink 里没有这个配置:  taskmanager.cpu.core  默认是   1   

另外:blink 里 taskmanager.heap.mb   与 flink 的 taskmanager.heap.size  不同导致   
taskmanager.heap 配置过小,默认1G

之前错误的配置:
# The heap size for the JobManager JVM

jobmanager.heap.size: 20480m


# The heap size for the TaskManager JVM

taskmanager.heap.size: 40960m


# The number of task slots that each TaskManager offers. Each slot runs one 
parallel pipeline.

taskmanager.numberOfTaskSlots: 24



现在配置:


# The heap size for the JobManager JVM

jobmanager.heap.mb: 20480


# The heap size for the TaskManager JVM

taskmanager.heap.mb: 102400


# How many physical cpu cores a task manager will supply for user

taskmanager.cpu.core: 30


# The number of task slots that each TaskManager offers. Each slot runs one 
parallel pipeline.

taskmanager.numberOfTaskSlots: 30



之前安装了Flink 1.7.2,后又安装 了 Blink,我直接拿的Flink 的配置,没注意看,在此希望我的经历能给大家一些提醒,谢谢。 

在此 感谢龙三 的帮忙,和  谢谢大家的回复。。

邓成刚【qq】
 
发件人: 邓成刚【qq】
发送时间: 2019-03-29 17:18
收件人: user-zh
主题: Re: Re: 请教一下Blink资源分配问题
用的是table , LOG里资源申请情况:
 
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request = 
5ac9229acae1e6ef90563a5a0bf3fe21).
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request = 
2e39ec9bb22b2afe2baa15a87d854796).
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3 with 
allocation id 1a8a3b10d3c235dcb64b4b91e0a22bc8.
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} from resource manager (request = 
ce1a10fca1c7a9f6f93dcd248ebce56c).
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3 with 
allocation id b9ea06e5ba61013ead29424b72e8535c.
ResourceProfile{cpuCores=0.6, heapMemoryInMB=96, directMemoryInMB=0, 
nativeMemoryInMB=0, networkMemoryInMB=2, MANAGED_MEMORY_MB=0.0, 
FLOATING_MANAGED_MEMORY_MB=0.0} for job e40415828bbe184dc051e422e85e98c3 with 
allocation id c3742a90378a264341e8d1a573c67535.
 
 
发件人: Guowei Ma
发送时间: 2019-03-29 17:12
收件人: user-zh
主题: Re: Re: 请教一下Blink资源分配问题
用的什么api,DataStream还是Table?
如果是DataStream的话,申请什么样的资源?
详细列下。
 
Best,
Guowei
 
 
邓成刚【qq】  于2019年3月29日周五 下午5:09写道:
 
> 是的。
>
> 发件人: moxian
> 发送时间: 2019-03-29 17:06
> 收件人: user-zh
> 主题: Re: 请教一下Blink资源分配问题
> standalone 模式?
>
> 邓成刚【qq】  于2019年3月29日周五 上午9:59写道:
>
> > 请教一下Blink资源分配问题:
> > blink 任务并行度设置 20  提示0个满足:Batch request 40 slots, but only 0 are
> fulfilled.
> > 调整到 3 并行度 提示:Batch request 6 slots, but only 4 are fulfilled.,
> > 但是我的TASK SLOTS有配 48,没有其它任务,
> > 按理説没有资源问题啊,集群配置情况:
> >
> > 其它的都是默认配置:
> >
> > taskmanager.numberOfTaskSlots: 24
> >
> > jobmanager.heap.size: 20480m
> >
> > # The heap size for the TaskManager JVM
> >
> > taskmanager.heap.size: 40960m
> >
> >
> > 服务器 2 台:每台 48核,256G
> >
> >
> >