[
https://issues.apache.org/jira/browse/FLINK-32552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742453#comment-17742453
]
Fabio Wanner commented on FLINK-32552:
--------------------------------------
I dug a bit deeper and found the following:
Anytime when the error happens there is one job deployment with this exception:
{code:java}
org.apache.flink.runtime.rest.handler.RestHandlerException: No jobs included in
application.{code}
This job will be deployed again by the operator and will run without any
problems.
But one of the other jobs deployed in parallel will get mixed up with the
failed one and will not get running (using jar file of the failed one leading
to classNotFound or same jobId and/or same name)...
> Mixed up Flink session job deployments
> --------------------------------------
>
> Key: FLINK-32552
> URL: https://issues.apache.org/jira/browse/FLINK-32552
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Fabio Wanner
> Priority: Major
>
> *Context*
> In the scope of end-to-end tests we deploy all the Flink session jobs we have
> regularly in a staging environment. Some of the jobs are bundled together in
> one helm chart and therefore deployed at the same time. There are around 40
> individual Flink jobs (running on the same Flink session cluster). The
> session cluster is individual for each e2e test run. The problems described
> below happen scarcely (1 in ~ 50 run maybe).
> *Problem*
> Rarely the operator seems to "mix up" the deployments. This can be seen in
> the Flink cluster logs as multiple {{Received JobGraph submission '<JOB
> NAME>' (<JOB_ID>)}} logs are created from jobs with the same job_id. This
> results in errors such as:
> {{DuplicateJobSubmissionException}} or {{ClassNotFoundException.}}
> It' also visible in the FlinkSessionJob resource: status.jobStatus.jobName
> does not match the expected job name of the job being deployed (The job name
> is passed to the application via argument).
> So far we were unable to reliably reproduce the error.
> *Details*
> The following lines show the status of 3 jobs form the view point of the
> Flink cluster dashboard, and the FlinkSessionJob ressource:
>
> *aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615*
> Apache Flink Dashboard:
> * State: Restarting
> * ID: a7d36f3881f943a00000000000000002
> * Exceptions: Cannot load user class:
> aelps.pipelines.aletsch.smc.SMCUrlMapper
> FlinkSessionJob Ressource:
> * State: RUNNING
> * jobId: a1221c743367497b0000000000000002
> * uid: a1221c74-3367-497b-ad2f-8793ab23919d
>
> *aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615*
> Apache Flink Dashboard:
> * State: -
> * ID: -
> FlinkSessionJob Ressource:
> * State: UPGRADING
> * jobId: -
> * uid: a7d36f38-81f9-43a0-898f-19b950430e9d
> Flink K8s Operator:
> * Exceptions: DuplicateJobSubmissionException: Job has already been
> submitted.
>
> *aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615*
> Apache Flink Dashboard:
> * State: Running
> * ID: e692c2dfaa18441c0000000000000002
> * Exceptions: -
> FlinkSessionJob Ressource:
> * State: RUNNING
> * jobId: e692c2dfaa18441c0000000000000002
> * uid: e692c2df-aa18-441c-a352-88aefa9a3017
> As we can see the *aletsch_smc* job is presumably running according to the
> FlinkSessionJob resource, but crash-looping in the cluster and it has the
> jobID matching the uid of the resource of {*}aletsch_mat{*}. While
> *aletsch_mat* is not even running. The following logs also show some
> suspicious entries: There are several {{Received JobGraph submission}} from
> different jobs with the same jobID.
>
> *Logs*
> The logs are filtered by the 3 jobIds from above.
>
> JobID: a7d36f3881f943a00000000000000002
> {code:bash}
> Flink Cluster
> ...
> 023-07-06 10:23:50,552 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
> 2023-07-06 10:23:50 file:
> '/tmp/tm_10.0.11.159:6122-e9fadc/blobStorage/job_a7d36f3881f943a00000000000000002/blob_p-40c7a30adef8868254191d2cf2dbc4cb7ab46f0d-8a02a0583d91c5e8e6c94f378aa444c2'
> (valid JAR)
> 2023-07-06 10:23:50,522 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=4}]
> 2023-07-06 10:23:50,522 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=3}]
> 2023-07-06 10:23:50,522 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=2}]
> 2023-07-06 10:23:50,522 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=1}]
> 2023-07-06 10:23:50,512 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state RESTARTING to RUNNING.
> 2023-07-06 10:23:48,979 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Clearing resource requirements of job a7d36f3881f943a00000000000000002
> 2023-07-06 10:23:48,853 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=1}]
> 2023-07-06 10:23:48,853 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=2}]
> 2023-07-06 10:23:48,853 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=3}]
> 2023-07-06 10:23:48 file:
> '/tmp/tm_10.0.11.159:6122-e9fadc/blobStorage/job_a7d36f3881f943a00000000000000002/blob_p-40c7a30adef8868254191d2cf2dbc4cb7ab46f0d-8a02a0583d91c5e8e6c94f378aa444c2'
> (valid JAR)
> 2023-07-06 10:23:48,661 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
> 2023-07-06 10:23:48,583 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=4}]
> 2023-07-06 10:23:48,583 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=3}]
> 2023-07-06 10:23:48,583 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=2}]
> 2023-07-06 10:23:48,582 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=1}]
> 2023-07-06 10:23:48,573 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state RESTARTING to RUNNING.
> 2023-07-06 10:23:47,562 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission 'aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:47,518 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Clearing resource requirements of job a7d36f3881f943a00000000000000002
> 2023-07-06 10:23:47,517 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=1}]
> 2023-07-06 10:23:47,517 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=2}]
> 2023-07-06 10:23:47,516 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=3}]
> 2023-07-06 10:23:47,463 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:47,463 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job a7d36f3881f943a00000000000000002 is submitted.
> 2023-07-06 10:23:47,104 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state RUNNING to RESTARTING.
> 2023-07-06 10:23:46,804 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Offer
> reserved slots to the leader of job a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:46,804 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Establish
> JobManager connection for job a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:46,799 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful
> registration at job manager
> akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2 for job
> a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:46,577 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 221b24b50413805c9e35d7620b8a00b8 for job
> a7d36f3881f943a00000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:46,577 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 49d3c8cd1080bd38c0144c3d3cc597cd for job
> a7d36f3881f943a00000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:46,577 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 819f34cc8957066478fb4b3549367d24 for job
> a7d36f3881f943a00000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:46,574 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job
> a7d36f3881f943a00000000000000002 for job leader monitoring.
> 2023-07-06 10:23:46,570 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 36802a7de1487f3fb1b6a3b509bd5e20 for job
> a7d36f3881f943a00000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:46,560 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a7d36f3881f943a00000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=4}]
> 2023-07-06 10:23:46,556 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registered job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2
> for job a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:46,528 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registering job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_2
> for job a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:46,480 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002) switched from state CREATED to RUNNING.
> 2023-07-06 10:23:46,476 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Starting
> execution of job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002) under job master id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:46,466 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> failover strategy
> org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@62877000
> for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:46,079 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Running
> initialization on master for job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:46,059 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Found 0 checkpoints in
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a7d36f3881f943a00000000000000002-config-map'}.
> 2023-07-06 10:23:46,051 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Recovering checkpoints from
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a7d36f3881f943a00000000000000002-config-map'}.
> 2023-07-06 10:23:46,006 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> restart back off time strategy
> ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000,
> maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000,
> jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,987 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] -
> Initializing job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,966 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission
> 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,965 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission 'aletsch_mat_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,915 INFO
> org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Added
> JobGraph(jobId: a7d36f3881f943a00000000000000002) to
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
> 2023-07-06 10:23:45,859 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting
> job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,857 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a7d36f3881f943a00000000000000002).
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job a7d36f3881f943a00000000000000002 is submitted.
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job a7d36f3881f943a00000000000000002 is submitted.
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=a7d36f3881f943a00000000000000002.
> 2023-07-06 10:23:45,705 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job a7d36f3881f943a00000000000000002 is submitted.
> Flink Operator
> 2023-07-06 10:26:25,792 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:25:05,163 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:24:24,553 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:24:03,850 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:23:53,094 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:23:47,346 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> 2023-07-06 10:23:45,372 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-mat-staging-e5730831] Submitting job:
> a7d36f3881f943a00000000000000002 to session cluster.
> {code}
>
> JobID: a1221c743367497b0000000000000002
> {code:bash}
> Flink Cluster
> 2023-07-06 11:23:48,062 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job a1221c743367497b0000000000000002 (48548 bytes,
> checkpointDuration=107 ms, finalizationTime=33 ms).
> 2023-07-06 11:23:47,937 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CheckpointType{name='Checkpoint',
> sharingFilesStrategy=FORWARD_BACKWARD}) @ 1688635427922 for job
> a1221c743367497b0000000000000002.
> 2023-07-06 10:23:48,567 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Offer
> reserved slots to the leader of job a1221c743367497b0000000000000002.
> 2023-07-06 10:23:48,567 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Establish
> JobManager connection for job a1221c743367497b0000000000000002.
> 2023-07-06 10:23:48,567 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful
> registration at job manager
> akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7 for job
> a1221c743367497b0000000000000002.
> 2023-07-06 10:23:48,009 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request cae6932e2409d5fece3f6b4636e3c71a for job
> a1221c743367497b0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,003 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 8a57f3ecff07d300aebb33f6b3545aed for job
> a1221c743367497b0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,003 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 7a4a0cfd16eec4a1cb043cce5f989db0 for job
> a1221c743367497b0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,002 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job
> a1221c743367497b0000000000000002 for job leader monitoring.
> 2023-07-06 10:23:48,002 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 92cbc64513fa703e4acf28bbb3088a58 for job
> a1221c743367497b0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,999 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> a1221c743367497b0000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=4}]
> 2023-07-06 10:23:47,998 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registered job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7
> for job a1221c743367497b0000000000000002.
> 2023-07-06 10:23:47,953 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registering job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_7
> for job a1221c743367497b0000000000000002.
> 2023-07-06 10:23:47,922 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a1221c743367497b0000000000000002) switched from state CREATED to RUNNING.
> 2023-07-06 10:23:47,887 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Starting
> execution of job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a1221c743367497b0000000000000002) under job master id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:47,887 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> failover strategy
> org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@2222ba4d
> for aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,880 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Running
> initialization on master for job
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,872 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Found 0 checkpoints in
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a1221c743367497b0000000000000002-config-map'}.
> 2023-07-06 10:23:47,867 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Recovering checkpoints from
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-a1221c743367497b0000000000000002-config-map'}.
> 2023-07-06 10:23:47,832 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> restart back off time strategy
> ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000,
> maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000,
> jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for
> aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,832 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] -
> Initializing job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,820 INFO
> org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Added
> JobGraph(jobId: a1221c743367497b0000000000000002) to
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
> 2023-07-06 10:23:47,780 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting
> job 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,776 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission 'aletsch_smc_e5730831db8092adb12f5189c4c895ef3a268615'
> (a1221c743367497b0000000000000002).
> 2023-07-06 10:23:47,668 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=a1221c743367497b0000000000000002.
> 2023-07-06 10:23:47,668 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job a1221c743367497b0000000000000002 is submitted.
> Flink Operator
> 2023-07-06 10:23:48,007 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-smc-staging-e5730831] Submitted job:
> a1221c743367497b0000000000000002 to session cluster.
> 2023-07-06 10:23:47,505 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-smc-staging-e5730831] Submitting job:
> a1221c743367497b0000000000000002 to session cluster.
> 2023-07-06 10:23:45,416 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-smc-staging-e5730831] Submitting job:
> a1221c743367497b0000000000000002 to session cluster.
> {code}
> JobID: e692c2dfaa18441c0000000000000002
> {code:bash}
> Flink Cluster
> 2023-07-06 11:23:48,004 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
> checkpoint 1 for job e692c2dfaa18441c0000000000000002 (8194 bytes,
> checkpointDuration=125 ms, finalizationTime=28 ms).
> 2023-07-06 11:23:47,867 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CheckpointType{name='Checkpoint',
> sharingFilesStrategy=FORWARD_BACKWARD}) @ 1688635427851 for job
> e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:48,568 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Offer
> reserved slots to the leader of job e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:48,568 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Establish
> JobManager connection for job e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:48,568 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Successful
> registration at job manager
> akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6 for job
> e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:48,002 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 5e5a0e55fac280bf31abf29a20bce684 for job
> e692c2dfaa18441c0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,002 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 1cdbce54f4376a1df86430f97dab6858 for job
> e692c2dfaa18441c0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,002 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request 352db7288d0e4d1775d5f52dd14c769d for job
> e692c2dfaa18441c0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,001 INFO
> org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Add job
> e692c2dfaa18441c0000000000000002 for job leader monitoring.
> 2023-07-06 10:23:48,000 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive
> slot request bffed3e4a4c8573049a4119bd7e15f19 for job
> e692c2dfaa18441c0000000000000002 from resource manager with leader id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:48,998 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
> [] - Received resource requirements from job
> e692c2dfaa18441c0000000000000002:
> [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
> numberOfRequiredSlots=4}]
> 2023-07-06 10:23:47,998 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registered job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6
> for job e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:47,953 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Registering job manager
> aaa9331f70b07a195b5f09d57d1b4...@akka.tcp://flink@10.0.11.158:6123/user/rpc/jobmanager_6
> for job e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:47,851 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615
> (e692c2dfaa18441c0000000000000002) switched from state CREATED to RUNNING.
> 2023-07-06 10:23:47,845 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Starting
> execution of job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615'
> (e692c2dfaa18441c0000000000000002) under job master id
> aaa9331f70b07a195b5f09d57d1b40c5.
> 2023-07-06 10:23:47,844 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> failover strategy
> org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@7eeab246
> for aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,834 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Running
> initialization on master for job
> aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,825 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Found 0 checkpoints in
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-e692c2dfaa18441c0000000000000002-config-map'}.
> 2023-07-06 10:23:47,813 INFO
> org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStoreUtils [] -
> Recovering checkpoints from
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-e692c2dfaa18441c0000000000000002-config-map'}.
> 2023-07-06 10:23:47,782 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] - Using
> restart back off time strategy
> ExponentialDelayRestartBackoffTimeStrategy(initialBackoffMS=1000,
> maxBackoffMS=300000, backoffMultiplier=2.0, resetBackoffThresholdMS=3600000,
> jitterFactor=0.5, currentBackoffMS=1000, lastFailureTimestamp=0) for
> aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,781 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] -
> Initializing job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615'
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,774 INFO
> org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Added
> JobGraph(jobId: e692c2dfaa18441c0000000000000002) to
> KubernetesStateHandleStore{configMapName='flink-cluster-aelps-staging-e5730831-cluster-config-map'}.
> 2023-07-06 10:23:47,703 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting
> job 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615'
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,702 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
> JobGraph submission
> 'aletsch_wp_wafer_e5730831db8092adb12f5189c4c895ef3a268615'
> (e692c2dfaa18441c0000000000000002).
> 2023-07-06 10:23:47,650 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=e692c2dfaa18441c0000000000000002.
> 2023-07-06 10:23:47,650 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job e692c2dfaa18441c0000000000000002 is submitted.
> Flink Operator
> 2023-07-06 10:23:47,973 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitted job:
> e692c2dfaa18441c0000000000000002 to session cluster.
> 2023-07-06 10:23:47,505 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitting job:
> e692c2dfaa18441c0000000000000002 to session cluster.
> 2023-07-06 10:23:45,374 o.a.f.k.o.s.AbstractFlinkService [INFO
> ][aelps-staging/aletsch-wp-wafer-staging-e5730831] Submitting job:
> e692c2dfaa18441c0000000000000002 to session cluster.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
