Thanks for sharing this,
The sequence of events the log seems strange to me:

2021-10-17 03:05:55,801 INFO
org.apache.flink.runtime.jobmaster.JobMaster                 [] -
Close ResourceManager connection c1092812cfb2853a5576ffd78e346189:
Stopping JobMaster for job 'rt-match_12.4.5_8d48b21a'
(00000000000000000000000000000000).
2021-10-17 03:05:59,382 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -
Starting KubernetesApplicationClusterEntrypoint (Version: 1.14.0,
Scala: 2.12, Rev:460b386, Date:2021-09-22T08:39:40+02:00)
2021-10-17 03:06:00,251 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -
RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2021-10-17 03:06:04,355 ERROR
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector []
- Exception occurred while acquiring lock 'ConfigMapLock: flink-ns -
match-70958037-f414-4925-9d60-19e90d12abc0-restserver-leader
(ef5c2463-2d66-4dce-a023-4b8a50d7acff)'
io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException:
Unable to create ConfigMapLock
Caused by: io.fabric8.kubernetes.client.KubernetesClientException:
Operation: [create]  for kind: [ConfigMap]  with name:
[match-70958037-f414-4925-9d60-19e90d12abc0-restserver-leader]  in
namespace: [flink-ns]  failed.
Caused by: java.io.InterruptedIOException

It looks like KubernetesApplicationClusterEntrypoint is re-started in
the middle of shutdown and, as a result, the resources it (re)creates
aren't clean up.

Could you please also share Kubernetes logs and resource definitions
to validate the above assumption?

Regards,
Roman

On Mon, Oct 25, 2021 at 6:15 AM Hua Wei Chen <[email protected]> wrote:
>
> Hi all,
>
> We have Flink jobs run on batch mode and get the job status via 
> JobHandler.onJobExecuted()[1].
>
> Base on the thread[2], we expected the Configmaps will be cleaned up after 
> execution successfully.
>
> But we found some Configmaps not be cleanup after job success. On the other 
> hand, the Configmaps contents and the labels are removed.
>
> Here is one of the Configmaps.
>
> ```
> apiVersion: v1
> kind: ConfigMap
> metadata:
>   name: match-6370b6ab-de17-4c93-940e-0ce06d05a7b8-resourcemanager-leader
>   namespace: app-flink
>   selfLink: >-
>     
> /api/v1/namespaces/app-flink/configmaps/match-6370b6ab-de17-4c93-940e-0ce06d05a7b8-resourcemanager-leader
>   uid: 80c79c87-d6e2-4641-b13f-338c3d3154b0
>   resourceVersion: '578806788'
>   creationTimestamp: '2021-10-21T17:06:48Z'
>   annotations:
>     control-plane.alpha.kubernetes.io/leader: >-
>       
> {"holderIdentity":"3da40a4a-0346-49e5-8d18-b04a68239bf3","leaseDuration":15.000000000,"acquireTime":"2021-10-21T17:06:48.092264Z","renewTime":"2021-10-21T17:06:48.092264Z","leaderTransitions":0}
>   managedFields:
>     - manager: okhttp
>       operation: Update
>       apiVersion: v1
>       time: '2021-10-21T17:06:48Z'
>       fieldsType: FieldsV1
>       fieldsV1:
>         'f:metadata':
>           'f:annotations':
>             .: {}
>             'f:control-plane.alpha.kubernetes.io/leader': {}
> data: {}
> ```
>
>
> Our Flink apps run on ver. 1.14.0.
> Thanks!
>
> BR,
> Oscar
>
>
> Reference:
> [1] JobListener (Flink : 1.15-SNAPSHOT API) (apache.org)
> [2] 
> https://lists.apache.org/[email protected]:lte=1M:High%20availability%20data%20clean%20up%20
>

Reply via email to