[
https://issues.apache.org/jira/browse/FLINK-24555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated FLINK-24555:
---------------------------------
Description:
h2. Why
As I found some warn logs in our production flink jobs, like as follows(the
detail logs attached at the end)
{code:java}
2021-10-14 17:46:41 WARN - [main] -
org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying to
split key and value in configuration file
/data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_000001/flink-conf.yaml:11:
"pipeline.classpaths: "
{code}
I dig the flink code and found it put the empty list into configuration by
{{ConfigUtils.encodeCollectionToConfig}}.
h2. How
So it's better to ignore the empty collection to put in the configuration.
Because these configuration will be written to the flink-conf.yaml and Flink
jobmanager will read but fail to read the corresponding key's value as the warn
log described.
h2. Appendix
{code:java}
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: internal.jobgraph-path, job.graph
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: restart-strategy.failure-rate.max-failures-per-interval, 3
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: jobmanager.execution.failover-strategy, region
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: high-availability.cluster-id, application_1633896099002_1043646
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: jobmanager.rpc.address, localhost
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: execution.runtime-mode, AUTOMATIC
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: security.kerberos.fetch.delegation-token, false
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: execution.savepoint.ignore-unclaimed-state, false
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: parallelism.default, 2
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: taskmanager.numberOfTaskSlots, 1
2021-10-14 17:46:41 WARN - [main] -
org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying to
split key and value in configuration file
/data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_000001/flink-conf.yaml:11:
"pipeline.classpaths: "
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: restart-strategy.failure-rate.failure-rate-interval, 1 d
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: yarn.application.name, app_StreamEngine_Prod_jiandan_beat_rec_test_all
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: yarn.application.queue, talos.job_streaming
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: taskmanager.memory.process.size, 1728m
{code}
was:
h2. Why
As I found some warn logs in our production flink jobs, like as follows(the
detail logs attached at the end)
{code:java}
2021-10-14 17:46:41 WARN - [main] -
org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying to
split key and value in configuration file
{code}
I dig the flink code and found it put the empty list into configuration by
{{ConfigUtils.encodeCollectionToConfig}}.
h2. How
So it's better to ignore the empty collection to put in the configuration.
Because these configuration will be written to the flink-conf.yaml and Flink
jobmanager will read but fail to read the corresponding key's value as the warn
log described.
h2. Appendix
{code:java}
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: internal.jobgraph-path, job.graph
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: restart-strategy.failure-rate.max-failures-per-interval, 3
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: jobmanager.execution.failover-strategy, region
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: high-availability.cluster-id, application_1633896099002_1043646
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: jobmanager.rpc.address, localhost
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: execution.runtime-mode, AUTOMATIC
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: security.kerberos.fetch.delegation-token, false
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: execution.savepoint.ignore-unclaimed-state, false
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: parallelism.default, 2
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: taskmanager.numberOfTaskSlots, 1
2021-10-14 17:46:41 WARN - [main] -
org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying to
split key and value in configuration file
/data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_000001/flink-conf.yaml:11:
"pipeline.classpaths: "
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: restart-strategy.failure-rate.failure-rate-interval, 1 d
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: yarn.application.name, app_StreamEngine_Prod_jiandan_beat_rec_test_all
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: yarn.application.queue, talos.job_streaming
2021-10-14 17:46:41 INFO - [main] -
org.apache.flink.configuration.GlobalConfiguration(213) - Loading configuration
property: taskmanager.memory.process.size, 1728m
{code}
> Incorrectly put empty list to flink configuration
> -------------------------------------------------
>
> Key: FLINK-24555
> URL: https://issues.apache.org/jira/browse/FLINK-24555
> Project: Flink
> Issue Type: Bug
> Reporter: Junfan Zhang
> Priority: Major
>
> h2. Why
> As I found some warn logs in our production flink jobs, like as follows(the
> detail logs attached at the end)
> {code:java}
> 2021-10-14 17:46:41 WARN - [main] -
> org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying
> to split key and value in configuration file
> /data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_000001/flink-conf.yaml:11:
> "pipeline.classpaths: "
> {code}
> I dig the flink code and found it put the empty list into configuration by
> {{ConfigUtils.encodeCollectionToConfig}}.
> h2. How
> So it's better to ignore the empty collection to put in the configuration.
> Because these configuration will be written to the flink-conf.yaml and Flink
> jobmanager will read but fail to read the corresponding key's value as the
> warn log described.
> h2. Appendix
> {code:java}
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: internal.jobgraph-path, job.graph
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property:
> restart-strategy.failure-rate.max-failures-per-interval, 3
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: jobmanager.execution.failover-strategy, region
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: high-availability.cluster-id,
> application_1633896099002_1043646
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: execution.runtime-mode, AUTOMATIC
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: security.kerberos.fetch.delegation-token, false
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: parallelism.default, 2
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-10-14 17:46:41 WARN - [main] -
> org.apache.flink.configuration.GlobalConfiguration(186) - Error while trying
> to split key and value in configuration file
> /data8/yarn/local/usercache/pizza/appcache/application_1633896099002_1043646/container_e429_1633896099002_1043646_01_000001/flink-conf.yaml:11:
> "pipeline.classpaths: "
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: restart-strategy.failure-rate.failure-rate-interval,
> 1 d
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: yarn.application.name,
> app_StreamEngine_Prod_jiandan_beat_rec_test_all
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: yarn.application.queue, talos.job_streaming
> 2021-10-14 17:46:41 INFO - [main] -
> org.apache.flink.configuration.GlobalConfiguration(213) - Loading
> configuration property: taskmanager.memory.process.size, 1728m
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
