[
https://issues.apache.org/jira/browse/FLINK-41031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis-Mircea Ciupitu updated FLINK-41031:
------------------------------------------
Description:
h1. Problem
The operator only stops two components on shutdown: the JOSDK {{Operator}} (via
{{installShutdownHook}}) and {{OperatorHealthService}} (via its own shutdown
hook, registered only in {{run()}}). {{FlinkOperator.stop()}} calls just
{{operator.stop()}}.
Everything else the operator creates is never closed:
* {{FlinkConfigManager}}: a non-daemon scheduler for config cache cleanup and
the dynamic config watcher
* {{CanaryResourceManager}}: a non-daemon scheduled pool per controller
* {{FlinkResourceContextFactory}}: the {{Flink-RestClusterClient-IO}} pool
* The operator {{MetricRegistry}} and its reporters
The same gap breaks startup failures. If the operator fails to start, for
example when {{operator.start()}} throws because an informer cannot list a
watched namespace, the {{main}} thread dies but the JVM keeps running, held up
by the {{FlinkConfigManager}} scheduler thread. The health server was never
started, so the container stays {{Running}} with no operator inside until the
startup probe kills it (about 5 minutes with the Helm chart defaults).
{{FlinkOperatorWebhook}} has the same problem: it also creates a
{{FlinkConfigManager}}, so a startup failure (missing environment variable,
keystore that cannot be loaded, failed bind) leaves the JVM running without a
server. The chart defines no probes for the webhook container, so it is never
restarted, and with {{failurePolicy: Fail}} every create or update of a Flink
resource is rejected until someone restarts it by hand.
h1. Impact
* On pod termination, the config watcher and canary checks keep firing while
JOSDK drains in-flight reconciliations. In-flight REST calls on the IO pool are
cut off instead of drained. Metric reporters are never closed, so push-based
reporters can lose their last flush.
* A failed startup is not visible as a failure. The pod reports {{Running}}
instead of {{Error}} or {{CrashLoopBackOff}}, and every restart attempt costs
the full startup probe budget.
* {{FlinkOperator.stop()}} leaves non-daemon threads running, so an embedded or
test operator does not release its resources. In tests, every
{{FlinkConfigManager}} leaks a scheduler thread that keeps the manager and its
config cache reachable for the rest of the surefire fork.
was:
h1. Problem
The operator only stops two components on shutdown: the JOSDK {{Operator}} (via
{{installShutdownHook}}) and {{OperatorHealthService}} (via its own shutdown
hook, registered only in {{run()}}). {{FlinkOperator.stop()}} calls just
{{operator.stop()}} and is only used for tests, not as a shutdown hook for the
operator itself.
Everything else the operator creates is never closed:
* {{FlinkConfigManager}}: a non-daemon scheduler for config cache cleanup and
the dynamic config watcher
* {{CanaryResourceManager}}: a non-daemon scheduled pool per controller
* {{FlinkResourceContextFactory}}: the {{Flink-RestClusterClient-IO}} pool
* The operator {{MetricRegistry}} and its reporters
h1. Impact
* On pod termination, the config watcher and canary checks keep firing while
JOSDK drains in-flight reconciliations. In-flight REST calls on the IO pool are
cut off instead of drained. Metric reporters are never closed, so push-based
reporters can lose their last flush.
* {{FlinkOperator.stop()}} leaves non-daemon threads running, so an embedded or
test operator does not release its resources. In tests, every
{{FlinkConfigManager}} leaks a scheduler thread that keeps the manager and its
config cache reachable for the rest of the surefire fork.
> Close operator-owned resources on shutdown and exit on startup failure
> ----------------------------------------------------------------------
>
> Key: FLINK-41031
> URL: https://issues.apache.org/jira/browse/FLINK-41031
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
>
> h1. Problem
> The operator only stops two components on shutdown: the JOSDK {{Operator}}
> (via {{installShutdownHook}}) and {{OperatorHealthService}} (via its own
> shutdown hook, registered only in {{run()}}). {{FlinkOperator.stop()}} calls
> just {{operator.stop()}}.
> Everything else the operator creates is never closed:
> * {{FlinkConfigManager}}: a non-daemon scheduler for config cache cleanup and
> the dynamic config watcher
> * {{CanaryResourceManager}}: a non-daemon scheduled pool per controller
> * {{FlinkResourceContextFactory}}: the {{Flink-RestClusterClient-IO}} pool
> * The operator {{MetricRegistry}} and its reporters
> The same gap breaks startup failures. If the operator fails to start, for
> example when {{operator.start()}} throws because an informer cannot list a
> watched namespace, the {{main}} thread dies but the JVM keeps running, held
> up by the {{FlinkConfigManager}} scheduler thread. The health server was
> never started, so the container stays {{Running}} with no operator inside
> until the startup probe kills it (about 5 minutes with the Helm chart
> defaults).
> {{FlinkOperatorWebhook}} has the same problem: it also creates a
> {{FlinkConfigManager}}, so a startup failure (missing environment variable,
> keystore that cannot be loaded, failed bind) leaves the JVM running without a
> server. The chart defines no probes for the webhook container, so it is never
> restarted, and with {{failurePolicy: Fail}} every create or update of a Flink
> resource is rejected until someone restarts it by hand.
> h1. Impact
> * On pod termination, the config watcher and canary checks keep firing while
> JOSDK drains in-flight reconciliations. In-flight REST calls on the IO pool
> are cut off instead of drained. Metric reporters are never closed, so
> push-based reporters can lose their last flush.
> * A failed startup is not visible as a failure. The pod reports {{Running}}
> instead of {{Error}} or {{CrashLoopBackOff}}, and every restart attempt costs
> the full startup probe budget.
> * {{FlinkOperator.stop()}} leaves non-daemon threads running, so an embedded
> or test operator does not release its resources. In tests, every
> {{FlinkConfigManager}} leaks a scheduler thread that keeps the manager and
> its config cache reachable for the rest of the surefire fork.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
