Martijn Visser created FLINK-40132:
--------------------------------------
Summary: MixedKubernetesServerExtension teardown flaky with
RejectedExecutionException from Fabric8 Vert.x mock server shutdown
Key: FLINK-40132
URL: https://issues.apache.org/jira/browse/FLINK-40132
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 2.4.0
Reporter: Martijn Visser
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77011&view=results
(leg: test_cron_azure misc)
05:07:40.216 [ERROR]
...InitTaskManagerDecoratorWithPodTemplateTest.testNodeSelectorMerging <<<
ERROR! java.util.concurrent.RejectedExecutionException: event executor
terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(...) at
io.vertx.core.impl.future.FutureImpl.onComplete(...) at
io.fabric8.mockwebserver.MockWebServer.await(MockWebServer.java:257) at
io.fabric8.mockwebserver.MockWebServer.shutdown(MockWebServer.java:169) at
io.fabric8.kubernetes.client.server.mock.KubernetesMockServer.destroy(...) at
org.apache.flink.kubernetes.MixedKubernetesServerExtension.afterEach(MixedKubernetesServerExtension.java:76)
{code}
Tests using the Fabric8 mock server intermittently fail in
{{MixedKubernetesServerExtension#afterEach}} ->
{{KubernetesMockServer.destroy()}} -> {{MockWebServer#shutdown()}} with
{{RejectedExecutionException: event executor terminated}}. The test body
passes; the exception is thrown during teardown (surefire attributes it to the
last-run method, here {{testNodeSelectorMerging}}).
This is an upstream Fabric8 bug: a timing race (+ non-idempotent shutdown) in
the Vert.x-based {{MockWebServer#shutdown()}} —
fabric8io/kubernetes-client#7747, fixed in Fabric8 7.7.0. It was introduced
into Flink by FLINK-38093, which moved the mock server onto the Vert.x
implementation when bumping Fabric8 to 7.3.1 (Flink master pins 7.3.1).
Fix: tolerate the RejectedExecutionException on teardown. Upgrading Fabric8 to
>= 7.7.0 is the eventual clean fix but drags OkHttp 3->5 and an unrelocated
Kotlin runtime into the core distribution, so it is deferred to a separate
dependency-upgrade ticket + dev@ discussion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
