Thanks for sharing this, The sequence of events the log seems strange to me:
2021-10-17 03:05:55,801 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Close ResourceManager connection c1092812cfb2853a5576ffd78e346189: Stopping JobMaster for job 'rt-match_12.4.5_8d48b21a' (00000000000000000000000000000000). 2021-10-17 03:05:59,382 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint (Version: 1.14.0, Scala: 2.12, Rev:460b386, Date:2021-09-22T08:39:40+02:00) 2021-10-17 03:06:00,251 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested. 2021-10-17 03:06:04,355 ERROR io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector [] - Exception occurred while acquiring lock 'ConfigMapLock: flink-ns - match-70958037-f414-4925-9d60-19e90d12abc0-restserver-leader (ef5c2463-2d66-4dce-a023-4b8a50d7acff)' io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: Unable to create ConfigMapLock Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [ConfigMap] with name: [match-70958037-f414-4925-9d60-19e90d12abc0-restserver-leader] in namespace: [flink-ns] failed. Caused by: java.io.InterruptedIOException It looks like KubernetesApplicationClusterEntrypoint is re-started in the middle of shutdown and, as a result, the resources it (re)creates aren't clean up. Could you please also share Kubernetes logs and resource definitions to validate the above assumption? Regards, Roman On Mon, Oct 25, 2021 at 6:15 AM Hua Wei Chen <[email protected]> wrote: > > Hi all, > > We have Flink jobs run on batch mode and get the job status via > JobHandler.onJobExecuted()[1]. > > Base on the thread[2], we expected the Configmaps will be cleaned up after > execution successfully. > > But we found some Configmaps not be cleanup after job success. On the other > hand, the Configmaps contents and the labels are removed. > > Here is one of the Configmaps. > > ``` > apiVersion: v1 > kind: ConfigMap > metadata: > name: match-6370b6ab-de17-4c93-940e-0ce06d05a7b8-resourcemanager-leader > namespace: app-flink > selfLink: >- > > /api/v1/namespaces/app-flink/configmaps/match-6370b6ab-de17-4c93-940e-0ce06d05a7b8-resourcemanager-leader > uid: 80c79c87-d6e2-4641-b13f-338c3d3154b0 > resourceVersion: '578806788' > creationTimestamp: '2021-10-21T17:06:48Z' > annotations: > control-plane.alpha.kubernetes.io/leader: >- > > {"holderIdentity":"3da40a4a-0346-49e5-8d18-b04a68239bf3","leaseDuration":15.000000000,"acquireTime":"2021-10-21T17:06:48.092264Z","renewTime":"2021-10-21T17:06:48.092264Z","leaderTransitions":0} > managedFields: > - manager: okhttp > operation: Update > apiVersion: v1 > time: '2021-10-21T17:06:48Z' > fieldsType: FieldsV1 > fieldsV1: > 'f:metadata': > 'f:annotations': > .: {} > 'f:control-plane.alpha.kubernetes.io/leader': {} > data: {} > ``` > > > Our Flink apps run on ver. 1.14.0. > Thanks! > > BR, > Oscar > > > Reference: > [1] JobListener (Flink : 1.15-SNAPSHOT API) (apache.org) > [2] > https://lists.apache.org/[email protected]:lte=1M:High%20availability%20data%20clean%20up%20 >
