[
https://issues.apache.org/jira/browse/FLINK-40132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-40132:
-----------------------------------
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77011&view=results
(leg: test_cron_azure misc)
{code}
05:07:40.216 [ERROR]
...InitTaskManagerDecoratorWithPodTemplateTest.testNodeSelectorMerging <<<
ERROR! java.util.concurrent.RejectedExecutionException: event executor
terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(...) at
io.vertx.core.impl.future.FutureImpl.onComplete(...) at
io.fabric8.mockwebserver.MockWebServer.await(MockWebServer.java:257) at
io.fabric8.mockwebserver.MockWebServer.shutdown(MockWebServer.java:169) at
io.fabric8.kubernetes.client.server.mock.KubernetesMockServer.destroy(...) at
org.apache.flink.kubernetes.MixedKubernetesServerExtension.afterEach(MixedKubernetesServerExtension.java:76)
{code}
Tests using the Fabric8 mock server intermittently fail in
{{MixedKubernetesServerExtension#afterEach}} ->
{{KubernetesMockServer.destroy()}} -> {{MockWebServer#shutdown()}} with
{{RejectedExecutionException: event executor terminated}}. The test body
passes; the exception is thrown during teardown (surefire attributes it to the
last-run method, here {{testNodeSelectorMerging}}).
This is an upstream Fabric8 bug: a timing race (+ non-idempotent shutdown) in
the Vert.x-based {{MockWebServer#shutdown()}} —
fabric8io/kubernetes-client#7747, fixed in Fabric8 7.7.0. It was introduced
into Flink by FLINK-38093, which moved the mock server onto the Vert.x
implementation when bumping Fabric8 to 7.3.1 (Flink master pins 7.3.1).
We could either tolerate the RejectedExecutionException on teardown, or choose
upgrading Fabric8 to >= 7.7.0. That is the eventual clean fix but drags OkHttp
3->5 and an unrelocated Kotlin runtime into the core distribution by default.
was:
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77011&view=results
(leg: test_cron_azure misc)
05:07:40.216 [ERROR]
...InitTaskManagerDecoratorWithPodTemplateTest.testNodeSelectorMerging <<<
ERROR! java.util.concurrent.RejectedExecutionException: event executor
terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(...) at
io.vertx.core.impl.future.FutureImpl.onComplete(...) at
io.fabric8.mockwebserver.MockWebServer.await(MockWebServer.java:257) at
io.fabric8.mockwebserver.MockWebServer.shutdown(MockWebServer.java:169) at
io.fabric8.kubernetes.client.server.mock.KubernetesMockServer.destroy(...) at
org.apache.flink.kubernetes.MixedKubernetesServerExtension.afterEach(MixedKubernetesServerExtension.java:76)
{code}
Tests using the Fabric8 mock server intermittently fail in
{{MixedKubernetesServerExtension#afterEach}} ->
{{KubernetesMockServer.destroy()}} -> {{MockWebServer#shutdown()}} with
{{RejectedExecutionException: event executor terminated}}. The test body
passes; the exception is thrown during teardown (surefire attributes it to the
last-run method, here {{testNodeSelectorMerging}}).
This is an upstream Fabric8 bug: a timing race (+ non-idempotent shutdown) in
the Vert.x-based {{MockWebServer#shutdown()}} —
fabric8io/kubernetes-client#7747, fixed in Fabric8 7.7.0. It was introduced
into Flink by FLINK-38093, which moved the mock server onto the Vert.x
implementation when bumping Fabric8 to 7.3.1 (Flink master pins 7.3.1).
We could either tolerate the RejectedExecutionException on teardown, or choose
upgrading Fabric8 to >= 7.7.0. That is the eventual clean fix but drags OkHttp
3->5 and an unrelocated Kotlin runtime into the core distribution by default.
> MixedKubernetesServerExtension teardown flaky with RejectedExecutionException
> from Fabric8 Vert.x mock server shutdown
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40132
> URL: https://issues.apache.org/jira/browse/FLINK-40132
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
> Affects Versions: 2.4.0
> Reporter: Martijn Visser
> Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77011&view=results
> (leg: test_cron_azure misc)
> {code}
> 05:07:40.216 [ERROR]
> ...InitTaskManagerDecoratorWithPodTemplateTest.testNodeSelectorMerging <<<
> ERROR! java.util.concurrent.RejectedExecutionException: event executor
> terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(...)
> at io.vertx.core.impl.future.FutureImpl.onComplete(...) at
> io.fabric8.mockwebserver.MockWebServer.await(MockWebServer.java:257) at
> io.fabric8.mockwebserver.MockWebServer.shutdown(MockWebServer.java:169) at
> io.fabric8.kubernetes.client.server.mock.KubernetesMockServer.destroy(...) at
> org.apache.flink.kubernetes.MixedKubernetesServerExtension.afterEach(MixedKubernetesServerExtension.java:76)
> {code}
> Tests using the Fabric8 mock server intermittently fail in
> {{MixedKubernetesServerExtension#afterEach}} ->
> {{KubernetesMockServer.destroy()}} -> {{MockWebServer#shutdown()}} with
> {{RejectedExecutionException: event executor terminated}}. The test body
> passes; the exception is thrown during teardown (surefire attributes it to
> the last-run method, here {{testNodeSelectorMerging}}).
> This is an upstream Fabric8 bug: a timing race (+ non-idempotent shutdown) in
> the Vert.x-based {{MockWebServer#shutdown()}} —
> fabric8io/kubernetes-client#7747, fixed in Fabric8 7.7.0. It was introduced
> into Flink by FLINK-38093, which moved the mock server onto the Vert.x
> implementation when bumping Fabric8 to 7.3.1 (Flink master pins 7.3.1).
> We could either tolerate the RejectedExecutionException on teardown, or
> choose upgrading Fabric8 to >= 7.7.0. That is the eventual clean fix but
> drags OkHttp 3->5 and an unrelocated Kotlin runtime into the core
> distribution by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
