gyfora commented on code in PR #1232:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/1232#discussion_r4443733516
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java:
##########
@@ -1243,6 +1247,74 @@ private static Configuration
getOperatorRestConfig(Configuration origConfig)
return conf;
}
+ @VisibleForTesting
+ protected void forceDeleteStuckPods(Map<String, String> labels, Deployment
deployment) {
+ var now = Instant.now();
+ var stuckPods =
+ kubernetesClient
+ .pods()
+ .inNamespace(deployment.getMetadata().getNamespace())
+ .withLabels(labels)
+ .list()
+ .getItems()
+ .stream()
+ .filter(pod -> isPastDeletionDeadline(pod, now))
+ .collect(Collectors.toList());
+ stuckPods.forEach(pod ->
kubernetesClient.resource(pod).withGracePeriod(0).delete());
+
+ if (!stuckPods.isEmpty()) {
+ EventUtils.createOrUpdateEventWithInterval(
+ kubernetesClient,
+ deployment,
+ EventRecorder.Type.Warning,
+ FORCE_DELETED_EVENT_KEY,
+ String.format(
+ "Force deleted %d pod(s) that did not terminate in
time "
+ + "(likely their node became unreachable),
blocking cluster resource cleanup.",
+ stuckPods.size()),
+ EventRecorder.Component.JobManagerDeployment,
+ event -> {},
+ FORCE_DELETED_EVENT_KEY,
+ null);
+ }
+ }
+
+ // The deletion timestamp of a pod already includes its termination grace
period
+ private static boolean isPastDeletionDeadline(Pod pod, Instant now) {
+ var deletionTimestamp = pod.getMetadata().getDeletionTimestamp();
+ return deletionTimestamp != null &&
now.isAfter(Instant.parse(deletionTimestamp));
+ }
+
+ protected Duration deleteDeploymentBlockingWithForceDelete(
+ String name,
+ Resource<Deployment> deployment,
+ DeletionPropagation propagation,
+ Duration timeout,
+ Map<String, String> podLabels,
+ Configuration conf) {
+ if (!conf.get(
+ KubernetesOperatorConfigOptions
+ .OPERATOR_RESOURCE_CLEANUP_FORCE_DELETE_STUCK_PODS)) {
Review Comment:
Not so sure, I don't really see a big downside of enabling this on a per CR
level. Unlike timeout settings this won't have an impact on the operator at
large
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
