[
https://issues.apache.org/jira/browse/FLINK-31974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719525#comment-17719525
]
Sergio Sainz commented on FLINK-31974:
--------------------------------------
Hi [~mapohl] - let me setup a new cluster later on to get the full logs. Below
please find the thread dump from the Flink 1.17.0 crash:
{code:java}
2023-04-28 20:50:50,305 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Received resource requirements from job 0a97c80a173b7ebb619c5b030b607520:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=1}]
...
2023-04-28 20:50:50,534 ERROR org.apache.flink.util.FatalExitExceptionHandler
[] - FATAL: Thread 'flink-akka.actor.default-dispatcher-15' produced
an uncaught exception. Stopping the process...
java.util.concurrent.CompletionException:
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException:
Failure executing: POST at:
https://10.96.0.1/api/v1/namespaces/env-my-namespace/pods. Message:
Forbidden!Configured service account doesn't have access. Service account may
have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-2" is
forbidden: exceeded quota: my-namespace-realtime-server-resource-quota,
requested: limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by:
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException:
Failure executing: POST at:
https://10.96.0.1/api/v1/namespaces/env-my-namespace/pods. Message:
Forbidden!Configured service account doesn't have access. Service account may
have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-2" is
forbidden: exceeded quota: my-namespace-realtime-server-resource-quota,
requested: limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:613)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
~[flink-dist-1.17.0.jar:1.17.0]
at
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163)
~[flink-dist-1.17.0.jar:1.17.0]
... 4 more
2023-04-28 20:50:50,602 ERROR org.apache.flink.util.FatalExitExceptionHandler
[] - Thread dump:
"main" prio=5 Id=1 WAITING on
java.util.concurrent.CompletableFuture$Signaller@2897b146
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.CompletableFuture$Signaller@2897b146
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.CompletableFuture$Signaller.block(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.CompletableFuture.waitingGet(Unknown
Source)
at java.base@11.0.19/java.util.concurrent.CompletableFuture.get(Unknown
Source)
at
app//org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:741)
at
app//org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint.main(KubernetesSessionClusterEntrypoint.java:61)"Reference
Handler" daemon prio=10 Id=2 RUNNABLE
at
java.base@11.0.19/java.lang.ref.Reference.waitForReferencePendingList(Native
Method)
at
java.base@11.0.19/java.lang.ref.Reference.processPendingReferences(Unknown
Source)
at
java.base@11.0.19/java.lang.ref.Reference$ReferenceHandler.run(Unknown
Source)"Finalizer" daemon prio=8 Id=3 WAITING on
java.lang.ref.ReferenceQueue$Lock@2b21a3e4
at java.base@11.0.19/java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@2b21a3e4
at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source)
at
java.base@11.0.19/java.lang.ref.Finalizer$FinalizerThread.run(Unknown
Source)"Signal Dispatcher" daemon prio=9 Id=4 RUNNABLE"Common-Cleaner" daemon
prio=8 Id=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@16aab001
at java.base@11.0.19/java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@16aab001
at java.base@11.0.19/java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.base@11.0.19/jdk.internal.ref.CleanerImpl.run(Unknown Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown Source)
at java.base@11.0.19/jdk.internal.misc.InnocuousThread.run(Unknown
Source)"Log4j2-TF-3-Scheduled-1" daemon prio=5 Id=12 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ee9ffbd
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ee9ffbd
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."Log4j2-TF-6-Scheduled-2" daemon prio=5 Id=18 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6166a7
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6166a7
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."flink-scheduler-1" prio=5 Id=19 TIMED_WAITING
at java.base@11.0.19/java.lang.Thread.sleep(Native Method)
at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-akka.actor.internal-dispatcher-2" prio=5 Id=20 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@1d79ccac
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@1d79ccac
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-akka.actor.default-dispatcher-4" prio=5 Id=22 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-akka.remote.default-remote-dispatcher-5" prio=5 Id=23 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@3fe545c2
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@3fe545c2
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-akka.remote.default-remote-dispatcher-6" prio=5 Id=24
TIMED_WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@3fe545c2
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@3fe545c2
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkUntil(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"New I/O worker #1" prio=5 Id=25 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@f0148c7
- locked sun.nio.ch.EPollSelectorImpl@5c4ee927
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@708d421"New I/O boss
#2" prio=5 Id=27 RUNNABLE
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@6f9289
- locked sun.nio.ch.EPollSelectorImpl@15b10c13
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@409550d1"New I/O
worker #3" prio=5 Id=28 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@10733912
- locked sun.nio.ch.EPollSelectorImpl@3bdffdea
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@dbc95df"New I/O server
boss #4" prio=5 Id=29 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@4758dbc2
- locked sun.nio.ch.EPollSelectorImpl@5a1496ae
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@85887aa"Timer-0"
daemon prio=5 Id=32 TIMED_WAITING on java.util.TaskQueue@53a69f57
at java.base@11.0.19/java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@53a69f57
at java.base@11.0.19/java.util.TimerThread.mainLoop(Unknown Source)
at java.base@11.0.19/java.util.TimerThread.run(Unknown Source)"BLOB
Server listener at 6124" daemon prio=5 Id=31 RUNNABLE (in native)
at java.base@11.0.19/java.net.PlainSocketImpl.socketAccept(Native
Method)
at java.base@11.0.19/java.net.AbstractPlainSocketImpl.accept(Unknown
Source)
at java.base@11.0.19/java.net.ServerSocket.implAccept(Unknown Source)
at java.base@11.0.19/java.net.ServerSocket.accept(Unknown Source)
at
app//org.apache.flink.util.NetUtils.acceptWithoutTimeout(NetUtils.java:143)
at
app//org.apache.flink.runtime.blob.BlobServer.run(BlobServer.java:316)"flink-metrics-scheduler-1"
prio=5 Id=34 TIMED_WAITING
at java.base@11.0.19/java.lang.Thread.sleep(Native Method)
at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-metrics-akka.actor.internal-dispatcher-3" prio=5 Id=36 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@17521864
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@17521864
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-metrics-akka.remote.default-remote-dispatcher-5" prio=5 Id=38
TIMED_WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@16555787
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@16555787
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkUntil(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-metrics-akka.remote.default-remote-dispatcher-6" prio=5 Id=39
WAITING on akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@16555787
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@16555787
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"New I/O worker #5" prio=5 Id=40 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@688a0205
- locked sun.nio.ch.EPollSelectorImpl@f316464
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@7d652799"New I/O boss
#6" prio=5 Id=42 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@827ccde
- locked sun.nio.ch.EPollSelectorImpl@34f25b32
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@73a2c645"New I/O
worker #7" prio=5 Id=43 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@cd0fd83
- locked sun.nio.ch.EPollSelectorImpl@846aa39
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
... Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@779d4ce9"New I/O
server boss #8" prio=5 Id=44 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@3eb1c6d3
- locked sun.nio.ch.EPollSelectorImpl@18bc21fc
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
... Number of locked synchronizers = 1
-
java.util.concurrent.ThreadPoolExecutor$Worker@329fcc7e"flink-rest-server-netty-boss-thread-1"
daemon prio=5 Id=48 RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked
org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySet@130bfb8d
- locked sun.nio.ch.EPollSelectorImpl@2ea19cb7
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526)
at
app//org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
..."Flink-DispatcherRestEndpoint-thread-1" daemon prio=5 Id=49
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."cluster-io-thread-1" daemon prio=5 Id=50 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"pool-2-thread-1" prio=5 Id=51 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bc769e4
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bc769e4
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"cluster-io-thread-2" daemon prio=5 Id=52 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-akka.actor.default-dispatcher-13" prio=5 Id=55 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"flink-akka.actor.default-dispatcher-14" prio=5 Id=56 WAITING on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool@5e904dba
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ForkJoinWorkerThread.run(Unknown
Source)"resourcemanager_1-main-scheduler-thread-1" daemon prio=5 Id=73
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9a11c0f
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9a11c0f
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."flink-kubeclient-io-for-resourcemanager-thread-1" daemon prio=5
Id=74 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-akka.actor.default-dispatcher-15" prio=5 Id=75 RUNNABLE
at
java.management@11.0.19/sun.management.ThreadImpl.dumpThreads0(Native Method)
at
java.management@11.0.19/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at
java.management@11.0.19/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at
app//org.apache.flink.util.concurrent.ThreadUtils.errorLogThreadDump(ThreadUtils.java:33)
at
app//org.apache.flink.util.FatalExitExceptionHandler.uncaughtException(FatalExitExceptionHandler.java:47)
at
app//org.apache.flink.util.concurrent.FutureUtils.lambda$handleUncaughtException$20(FutureUtils.java:1216)
at
app//org.apache.flink.util.concurrent.FutureUtils$$Lambda$623/0x00000001006a9840.accept(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source)
..."OkHttp https://10.96.0.1/..."; prio=5 Id=76 RUNNABLE (in native)
at java.base@11.0.19/java.net.SocketInputStream.socketRead0(Native
Method)
at java.base@11.0.19/java.net.SocketInputStream.socketRead(Unknown
Source)
at java.base@11.0.19/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.19/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.read(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown
Source)
... Number of locked synchronizers = 2
- java.util.concurrent.locks.ReentrantLock$NonfairSync@234e456e
- java.util.concurrent.ThreadPoolExecutor$Worker@330607e6"OkHttp
ConnectionPool" daemon prio=5 Id=77 TIMED_WAITING on
org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool@335680aa
at java.base@11.0.19/java.lang.Object.wait(Native Method)
- waiting on
org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool@335680aa
at java.base@11.0.19/java.lang.Object.wait(Unknown Source)
at
app//org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool.lambda$new$0(RealConnectionPool.java:62)
at
app//org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool$$Lambda$709/0x000000010071d040.run(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown Source) Number
of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@457b7a36"Okio
Watchdog" daemon prio=5 Id=78 TIMED_WAITING on java.lang.Class@711498ff
at java.base@11.0.19/java.lang.Object.wait(Native Method)
- waiting on java.lang.Class@711498ff
at
app//org.apache.flink.kubernetes.shaded.okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:348)
at
app//org.apache.flink.kubernetes.shaded.okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:313)"flink-kubeclient-io-for-resourcemanager-thread-2"
daemon prio=5 Id=80 RUNNABLE
at
java.base@11.0.19/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base@11.0.19/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at
java.base@11.0.19/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base@11.0.19/java.lang.reflect.Method.invoke(Unknown Source)
at
app//org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:689)
at
app//org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
at
app//org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at
app//org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
... Number of locked synchronizers = 1
-
java.util.concurrent.ThreadPoolExecutor$Worker@5fdf1d80"cluster-io-thread-3"
daemon prio=5 Id=84 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"Flink-DispatcherRestEndpoint-thread-2" daemon prio=5 Id=85 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."flink-rest-server-netty-worker-thread-1" daemon prio=5 Id=86
RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked
org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySet@3296b3ab
- locked sun.nio.ch.EPollSelectorImpl@72ad4a98
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879)
at
app//org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526)
at
app//org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
..."Flink-DispatcherRestEndpoint-thread-3" daemon prio=5 Id=88 WAITING
on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."Flink-DispatcherRestEndpoint-thread-4" daemon prio=5 Id=89
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e9640ee
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."OkHttp WebSocket https://10.96.0.1/..."; prio=5 Id=98 TIMED_WAITING
on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23991496
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23991496
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."jobmanager-io-thread-1" daemon prio=5 Id=99 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24cd939f
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24cd939f
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-akka.actor.supervisor-dispatcher-16" prio=5 Id=100 TIMED_WAITING
on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23417746
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23417746
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"flink-metrics-13" prio=1 Id=102 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ba0103c
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ba0103c
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"cluster-io-thread-4" daemon prio=5 Id=108 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67dc2101
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"jobmanager-future-thread-1" daemon prio=5 Id=109 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50769e7a
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50769e7a
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."flink-kubeclient-io-for-resourcemanager-thread-3" daemon prio=5
Id=112 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown Source)"OkHttp
10.96.0.1 Writer" prio=5 Id=113 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186fae80
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186fae80
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."OkHttp 10.96.0.1" daemon prio=5 Id=114 RUNNABLE (in native)
at java.base@11.0.19/java.net.SocketInputStream.socketRead0(Native
Method)
at java.base@11.0.19/java.net.SocketInputStream.socketRead(Unknown
Source)
at java.base@11.0.19/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.19/java.net.SocketInputStream.read(Unknown Source)
at java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.read(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown
Source)
at
java.base@11.0.19/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown
Source)
... Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@303c118c"OkHttp
Http2Connection" daemon prio=5 Id=115 TIMED_WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@ba6d647
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.SynchronousQueue$TransferStack@ba6d647
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown
Source)
at java.base@11.0.19/java.util.concurrent.SynchronousQueue.poll(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."FlinkCompletableFutureDelayScheduler-thread-1" daemon prio=5 Id=117
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6fc71c07
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6fc71c07
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."flink-kubeclient-io-for-resourcemanager-thread-4" daemon prio=5
Id=118 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@251cace5
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1" daemon
prio=5 Id=119 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ab9b4d1
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ab9b4d1
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base@11.0.19/java.lang.Thread.run(Unknown
Source)"jobmanager_3-main-scheduler-thread-1" daemon prio=5 Id=121
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a59647b
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a59647b
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."SourceCoordinator-Source: *anonymous_kafka$15*[66]" prio=5 Id=122
WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67708599
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67708599
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."Checkpoint Timer" daemon prio=5 Id=123 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30809a54
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30809a54
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.parkNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
..."kafka-admin-client-thread |
KafkaSource--5297459524620964018-enumerator-admin-client" daemon prio=5 Id=125
RUNNABLE (in native)
at java.base@11.0.19/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11.0.19/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown
Source)
at java.base@11.0.19/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
Source)
- locked sun.nio.ch.Util$2@77d8bfa9
- locked sun.nio.ch.EPollSelectorImpl@66b37795
at java.base@11.0.19/sun.nio.ch.SelectorImpl.select(Unknown Source)
at
app//org.apache.kafka.common.network.Selector.select(Selector.java:873)
at app//org.apache.kafka.common.network.Selector.poll(Selector.java:465)
at
app//org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
at
app//org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1408)
..."SourceCoordinator-Source: *anonymous_kafka$15*[66]-worker-thread-1"
daemon prio=5 Id=127 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58c6a97
at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58c6a97
at
java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at
java.base@11.0.19/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
...
2023-04-28 20:50:50,607 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting
KubernetesSessionClusterEntrypoint down with application status UNKNOWN.
Diagnostics Cluster entrypoint has been closed externally..
2023-04-28 20:50:50,609 INFO org.apache.flink.runtime.blob.BlobServer
[] - Stopped BLOB server at 0.0.0.0:6124
[pod@my-namespace-flink-cluster-59cd575d6b-bll6b log]$ {code}
> JobManager crashes after KubernetesClientException exception with
> FatalExitExceptionHandler
> -------------------------------------------------------------------------------------------
>
> Key: FLINK-31974
> URL: https://issues.apache.org/jira/browse/FLINK-31974
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
> Affects Versions: 1.17.0
> Reporter: Sergio Sainz
> Assignee: Weijie Guo
> Priority: Major
>
> When resource quota limit is reached JobManager will throw
>
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException:
> Failure executing: POST at:
> https://10.96.0.1/api/v1/namespaces/my-namespace/pods. Message:
> Forbidden!Configured service account doesn't have access. Service account may
> have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-2" is
> forbidden: exceeded quota: my-namespace-resource-quota, requested:
> limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
>
> In {*}1.16.1 , this is handled gracefully{*}:
> {code}
> 2023-04-28 22:07:24,631 WARN
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Failed requesting worker with resource spec WorkerResourceSpec
> \{cpuCores=1.0, taskHeapSize=25.600mb (26843542 bytes), taskOffHeapSize=0
> bytes, networkMemSize=64.000mb (67108864 bytes), managedMemSize=230.400mb
> (241591914 bytes), numSlots=4}, current pending count: 0
> java.util.concurrent.CompletionException:
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at: https://10.96.0.1/api/v1/namespaces/my-namespace/pods. Message:
> Forbidden!Configured service account doesn't have access. Service account may
> have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-138" is
> forbidden: exceeded quota: my-namespace-resource-quota, requested:
> limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
> at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> Source) ~[?:?]
> at java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> Source) ~[?:?]
> at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown
> Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> ~[?:?]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: POST at: https://10.96.0.1/api/v1/namespaces/my-namespace/pods.
> Message: Forbidden!Configured service account doesn't have access. Service
> account may have been revoked. pods
> "my-namespace-flink-cluster-taskmanager-1-138" is forbidden: exceeded quota:
> my-namespace-resource-quota, requested: limits.cpu=3, used:
> limits.cpu=12100m, limited: limits.cpu=13.
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:613)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163)
> ~[flink-dist-1.16.1.jar:1.16.1]
> ... 4 more
> {code}
> But , {*}in Flink 1.17.0 , Job Manager crashes{*}:
> {code}
> 2023-04-28 20:50:50,534 ERROR org.apache.flink.util.FatalExitExceptionHandler
> [] - FATAL: Thread 'flink-akka.actor.default-dispatcher-15'
> produced an uncaught exception. Stopping the process...
> java.util.concurrent.CompletionException:
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException:
> Failure executing: POST at:
> https://10.96.0.1/api/v1/namespaces/my-namespace/pods. Message:
> Forbidden!Configured service account doesn't have access. Service account may
> have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-2" is
> forbidden: exceeded quota: my-namespace-resource-quota, requested:
> limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
> at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> Source) ~[?:?]
> at java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> Source) ~[?:?]
> at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown
> Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> ~[?:?]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
> Caused by:
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException:
> Failure executing: POST at:
> https://10.96.0.1/api/v1/namespaces/my-namespace/pods. Message:
> Forbidden!Configured service account doesn't have access. Service account may
> have been revoked. pods "my-namespace-flink-cluster-taskmanager-1-2" is
> forbidden: exceeded quota: my-namespace-resource-quota, requested:
> limits.cpu=3, used: limits.cpu=12100m, limited: limits.cpu=13.
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:613)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[flink-dist-1.17.0.jar:1.17.0]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163)
> ~[flink-dist-1.17.0.jar:1.17.0]
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
