[
https://issues.apache.org/jira/browse/FLINK-41267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santwana Verma updated FLINK-41267:
-----------------------------------
Description:
For a FlinkDeployment whose job has checkpointing disabled, which includes
every batch job, the operator logs the following with a full stack trace on
every reconcile:
{{WARN Exception while fetching cluster info
java.util.concurrent.ExecutionException:
org.apache.flink.runtime.rest.util.RestClientException: [Checkpointing has not
been enabled.]}}
status.clusterInfo and status.taskManager.replicas are also never updated for
these jobs.
The cluster info includes the job's state size, which the operator reads from
the job's checkpoint statistics. For a job without checkpointing, the
JobManager answers 404 "Checkpointing has not been enabled." The operator
treats that answer as a failure of the whole cluster info, so none of it is
reported. That request also has no timeout, unlike the operator's other
requests for the cluster info.
When the operator looks up a job's latest checkpoint, it already accepts the
same answer as "no checkpoint". It still logs it as a WARN with the stack
trace, though.
{*}Proposed fix{*}: when the JobManager gives this answer, report the cluster
info without the state size, and log at DEBUG. Add the operator's Flink client
timeout to the checkpoint statistics request. Log the latest-checkpoint case at
DEBUG without the stack trace.
was:
For a FlinkDeployment whose job has checkpointing disabled, which includes
every batch job, the operator logs the following with a full stack trace on
every reconcile:
WARN Exception while fetching cluster info
java.util.concurrent.ExecutionException:
org.apache.flink.runtime.rest.util.RestClientException: [Checkpointing has not
been enabled.]
status.clusterInfo and status.taskManager.replicas are also never updated for
these jobs.
To report the job's state size in the cluster info,
AbstractFlinkService.getClusterInfo requests the job's checkpoint statistics.
For a job without checkpointing, the JobManager answers 404 "Checkpointing has
not been enabled." populateStateSize does not handle this, so the whole
getClusterInfo call fails. That request also has no timeout, unlike the other
requests in getClusterInfo.
Proposed fix: when Flink gives this answer, skip the state size in
populateStateSize and log at DEBUG. Add the Flink client timeout to that
request, and log the same case in getLastCheckpoint at DEBUG without the stack
trace.
> Cluster info fails for jobs without checkpointing, logging a WARN on every
> reconcile
> ------------------------------------------------------------------------------------
>
> Key: FLINK-41267
> URL: https://issues.apache.org/jira/browse/FLINK-41267
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Santwana Verma
> Priority: Minor
>
> For a FlinkDeployment whose job has checkpointing disabled, which includes
> every batch job, the operator logs the following with a full stack trace on
> every reconcile:
> {{WARN Exception while fetching cluster info
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.rest.util.RestClientException: [Checkpointing has
> not been enabled.]}}
> status.clusterInfo and status.taskManager.replicas are also never updated for
> these jobs.
> The cluster info includes the job's state size, which the operator reads from
> the job's checkpoint statistics. For a job without checkpointing, the
> JobManager answers 404 "Checkpointing has not been enabled." The operator
> treats that answer as a failure of the whole cluster info, so none of it is
> reported. That request also has no timeout, unlike the operator's other
> requests for the cluster info.
> When the operator looks up a job's latest checkpoint, it already accepts the
> same answer as "no checkpoint". It still logs it as a WARN with the stack
> trace, though.
> {*}Proposed fix{*}: when the JobManager gives this answer, report the cluster
> info without the state size, and log at DEBUG. Add the operator's Flink
> client timeout to the checkpoint statistics request. Log the
> latest-checkpoint case at DEBUG without the stack trace.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
