[
https://issues.apache.org/jira/browse/FLINK-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901802#comment-16901802
]
shuwenjun edited comment on FLINK-13590 at 8/7/19 10:02 AM:
------------------------------------------------------------
Hi Yang，
You are right, it's right that the files are created when launching a new yarn
container. And it is better optimized to override taskmanager-conf.yaml. Now,
it could cause a more serious problem, that it will produce hundreds of
thousands of small files, because the RM keep retrying to apply for
taskmanager(container) when the source is enough. And the RM always don't get
some new correct taskmanagers.
Thank you for your comment.
There are some logs.
{code:java}
//代码占位符
2019-08-07 16:54:52,729 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy -
Opening proxy : hadoopxxx
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 1.
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 2.
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 3.
2019-08-07 16:55:03,675 INFO org.apache.flink.yarn.YarnResourceManager -
Received new container: container_xxx_01_007684 - Remaining pending container
requests: 3
2019-08-07 16:55:03,675 INFO org.apache.flink.yarn.YarnResourceManager -
Removing container request Capability[<memory:8192, vCores:2>]Priority[1].
Pending container requests 2.
2019-08-07 16:55:03,676 INFO org.apache.flink.yarn.YarnResourceManager - Adding
keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,740 INFO org.apache.flink.yarn.YarnResourceManager -
Creating container launch context for TaskManagers
2019-08-07 16:55:03,740 INFO org.apache.flink.yarn.YarnResourceManager -
Starting TaskManagers
2019-08-07 16:55:03,742 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy -
Opening proxy : hadoopxxx
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager -
Received new container: container_xxx_01_007685 - Remaining pending container
requests: 2
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager -
Removing container request Capability[<memory:8192, vCores:2>]Priority[1].
Pending container requests 1.
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager - Adding
keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,761 INFO org.apache.flink.yarn.YarnResourceManager -
Creating container launch context for TaskManagers
2019-08-07 16:55:03,761 INFO org.apache.flink.yarn.YarnResourceManager -
Starting TaskManagers
2019-08-07 16:55:03,761 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy -
Opening proxy : hadoopxxx
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager -
Received new container: container_xxx_01_007686 - Remaining pending container
requests: 1
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager -
Removing container request Capability[<memory:8192, vCores:2>]Priority[1].
Pending container requests 0.
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager - Adding
keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,814 INFO org.apache.flink.yarn.YarnResourceManager -
Creating container launch context for TaskManagers
2019-08-07 16:55:03,814 INFO org.apache.flink.yarn.YarnResourceManager -
Starting TaskManagers
2019-08-07 16:55:03,814 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy -
Opening proxy : hadoopxxx
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 1.
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 2.
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager -
Requesting new TaskExecutor container with resources <memory:8192, vCores:2>.
Number pending requests 3.
2019-08-07 16:55:14,682 INFO org.apache.flink.yarn.YarnResourceManager -
Received new container: container_xxx - Remaining pending container requests: 3
2019-08-07 16:55:14,683 INFO org.apache.flink.yarn.YarnResourceManager -
Removing container request Capability[<memory:8192, vCores:2>]Priority[1].
Pending container requests 2.
2019-08-07 16:55:14,683 INFO org.apache.flink.yarn.YarnResourceManager - Adding
keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:14,778 INFO org.apache.flink.yarn.YarnResourceManager -
Creating container launch context for TaskManagers
2019-08-07 16:55:14,778 INFO org.apache.flink.yarn.YarnResourceManager -
Starting TaskManagers
{code}
was (Author: [email protected]):
Hi Yang，
You are right, it's right that the files are created when launching a new yarn
container. And it is better optimized to override taskmanager-conf.yaml. Now,
it could cause a more serious problem, that it will produce hundreds of
thousands of small files, because the RM keep retrying to apply for
taskmanager(container) when the source is enough. And the RM always don't get
some new correct taskmanagers.
Thank you for your comment.
> flink-on-yarn sometimes could create many little files that are
> xxx-taskmanager-conf.yaml
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-13590
> URL: https://issues.apache.org/jira/browse/FLINK-13590
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Reporter: shuwenjun
> Priority: Major
> Attachments: taskmanager-conf-yaml.png
>
>
> Both of 1.7.2 and 1.8.0 are used, but they could create many little files.
> These files are the configuration file of taskmanager and when the flink
> session try to apply a new container, one of the files will be created. And I
> don't know why sometimes the flink session apply container again and again?
> Or when one container has lost, it could delete its taskmanager-conf.yaml
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
