[
https://issues.apache.org/jira/browse/FLINK-21647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413242#comment-17413242
]
Kevin commented on FLINK-21647:
-------------------------------
Hello [~wangyang0918], thanks for you reply!
My goal is to run Kafka 2.8.0 and Flink 1.13.2 on Amazone Kubernetes Service
1.20.7 (AKS). AKS seems to run fine and Kafka (which I deployed via Strimzi)
can produce and consume message via console. However, I now struggle with
staring a Flink session with following command:
{{}}
{code:java}
./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster{code}
{{Do the logging logs provide meangingful insights to you?}}
{{BR,}}
{{Kevin}}
----
{{Checking status of pods:}}
{code:java}
kevin@road-condition-vm-flink-client:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-first-flink-clustercd-6d59756c7c-9fb7s 0/1 CrashLoopBackOff 167 14h
road-condition-kafka-kafka-0 1/1 Running 0 11d
road-condition-kafka-zookeeper-0 1/1 Running 0 11d
strimzi-cluster-operator-687fdd6f77-24ccn 1/1 Running 7 11d
{code}
Checking status of services (flink monitoring service is not accessible even
both is provided, internal & external IP) – I have masked the external IP in
the code below:
{code:java}
kevin@road-condition-vm-flink-client:~$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 31d
my-first-flink-clustercd ClusterIP None <none> 6123/TCP,6124/TCP 14h
my-first-flink-clustercd-rest LoadBalancer 10.0.138.102 20.XX.XX.0
8081:32603/TCP 14h
road-condition-kafka-kafka-bootstrap ClusterIP 10.0.99.147 <none>
9091/TCP,9092/TCP 11d
road-condition-kafka-kafka-brokers ClusterIP None <none>
9090/TCP,9091/TCP,9092/TCP 11d
road-condition-kafka-zookeeper-client ClusterIP 10.0.59.247 <none> 2181/TCP 11d
road-condition-kafka-zookeeper-nodes ClusterIP None <none>
2181/TCP,2888/TCP,3888/TCP 11d{code}
Checking logs of flink session pod:
{code:java}
kevin@road-condition-vm-flink-client:~$ kubectl logs
my-first-flink-clustercd-6d59756c7c-9fb7s
sed: couldn't open temporary file /opt/flink/conf/sed0NT8SI: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sed5rHenH: Read-only file
system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedi357bL: Read-only file
system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
Starting kubernetes-session as a console application on host
my-first-flink-clustercd-6d59756c7c-9fb7s.
2021-09-10 15:14:04,524 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-09-10 15:14:04,527 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration:
2021-09-10 15:14:04,527 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456
dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D
jobmanager.memory.jvm-overhead.min=201326592b -D
jobmanager.memory.jvm-metaspace.size=268435456b -D
jobmanager.memory.heap.size=1073741824b -D
jobmanager.memory.jvm-overhead.max=201326592b
logs: INFO [] - Loading configuration property: blob.server.port, 6124
INFO [] - Loading configuration property: taskmanager.memory.process.size, 1728m
INFO [] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
INFO [] - Loading configuration property:
jobmanager.execution.failover-strategy, region
INFO [] - Loading configuration property: jobmanager.rpc.address,
my-first-flink-clustercd.default
INFO [] - Loading configuration property: execution.target, kubernetes-session
INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m
INFO [] - Loading configuration property: jobmanager.rpc.port, 6123
INFO [] - Loading configuration property: kubernetes.cluster-id,
my-first-flink-clustercd
INFO [] - Loading configuration property: taskmanager.rpc.port, 6122
INFO [] - Loading configuration property: internal.cluster.execution-mode,
NORMAL
INFO [] - Loading configuration property: parallelism.default, 1
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
INFO [] - The derived from fraction jvm overhead memory (160.000mb (167772162
bytes)) is less than its min value 192.000mb (201326592 bytes), min value will
be used instead
INFO [] - Final Master Memory configuration:
INFO [] - Total Process Memory: 1.563gb (1677721600 bytes)
INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes)
INFO [] - JVM Heap: 1024.000mb (1073741824 bytes)
INFO [] - Off-heap: 128.000mb (134217728 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 192.000mb (201326592 bytes)
2021-09-10 15:14:04,528 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-09-10 15:14:04,528 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesSessionClusterEntrypoint (Version: 1.13.2, Scala: 2.11, Rev:5f007ff,
Date:2021-07-23T04:35:55+02:00)
2021-09-10 15:14:04,529 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user:
flink
2021-09-10 15:14:04,529 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2021-09-10 15:14:04,529 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit
Server VM - Oracle Corporation - 1.8/25.302-b08
2021-09-10 15:14:04,529 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size:
989 MiBytes
2021-09-10 15:14:04,529 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
/usr/local/openjdk-8
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency
available
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-XX:MaxMetaspaceSize=268435456
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog.file=/opt/flink/log/flink--kubernetes-session-0-my-first-flink-clustercd-6d59756c7c-9fb7s.log
2021-09-10 15:14:04,530 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-09-10 15:14:04,531 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-09-10 15:14:04,531 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-09-10 15:14:04,531 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2021-09-10 15:14:04,532 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-09-10 15:14:04,532 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.off-heap.size=134217728b
2021-09-10 15:14:04,532 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-09-10 15:14:04,532 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.min=201326592b
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-metaspace.size=268435456b
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.heap.size=1073741824b
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.max=201326592b
2021-09-10 15:14:04,533 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
/opt/flink/lib/flink-csv-1.13.2.jar:/opt/flink/lib/flink-json-1.13.2.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.13.2.jar:/opt/flink/lib/flink-table_2.11-1.13.2.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.13.2.jar:::
2021-09-10 15:14:04,534 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-09-10 15:14:04,535 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX
signal handlers for [TERM, HUP, INT]
2021-09-10 15:14:04,547 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: blob.server.port, 6124
2021-09-10 15:14:04,547 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-09-10 15:14:04,547 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.execution.failover-strategy,
region
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.address,
my-first-flink-clustercd.default
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.target, kubernetes-session
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.port, 6123
2021-09-10 15:14:04,548 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.cluster-id,
my-first-flink-clustercd
2021-09-10 15:14:04,549 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.rpc.port, 6122
2021-09-10 15:14:04,549 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-09-10 15:14:04,549 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: parallelism.default, 1
2021-09-10 15:14:04,549 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-09-10 15:14:04,619 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesSessionClusterEntrypoint.
2021-09-10 15:14:04,659 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default
filesystem.
2021-09-10 15:14:04,709 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is
not in the classpath/dependencies. The extended set of supported File Systems
via Hadoop is not available.
2021-09-10 15:14:04,743 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security
context.
2021-09-10 15:14:04,754 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-09-10 15:14:04,758 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be
created as /tmp/jaas-5459458947931074872.conf.
2021-09-10 15:14:04,810 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2021-09-10 15:14:04,811 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster
services.
2021-09-10 15:14:04,833 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start
actor system, external address my-first-flink-clustercd.default:6123, bind
address 0.0.0.0:6123.
2021-09-10 15:14:05,635 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger
started
2021-09-10 15:14:05,708 INFO akka.remote.Remoting [] - Starting remoting
2021-09-10 15:14:05,921 INFO akka.remote.Remoting [] - Remoting started;
listening on addresses :[akka.tcp://[email protected]:6123]
2021-09-10 15:14:06,036 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started
at akka.tcp://[email protected]:6123
2021-09-10 15:14:06,116 INFO org.apache.flink.configuration.Configuration [] -
Config uses fallback configuration key 'jobmanager.rpc.address' instead of key
'rest.address'
2021-09-10 15:14:06,124 INFO org.apache.flink.runtime.blob.BlobServer [] -
Created BLOB server storage directory
/tmp/blobStore-951adff6-057f-4c4e-a15e-150904156cc0
2021-09-10 15:14:06,131 INFO org.apache.flink.runtime.blob.BlobServer [] -
Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max
backlog: 1000
2021-09-10 15:14:06,162 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter
configured, no metrics will be exposed/reported.
2021-09-10 15:14:06,207 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start
actor system, external address my-first-flink-clustercd.default:0, bind address
0.0.0.0:0.
2021-09-10 15:14:06,244 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger
started
2021-09-10 15:14:06,251 INFO akka.remote.Remoting [] - Starting remoting
2021-09-10 15:14:06,313 INFO akka.remote.Remoting [] - Remoting started;
listening on addresses
:[akka.tcp://[email protected]:36611]
2021-09-10 15:14:06,324 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started
at akka.tcp://[email protected]:36611
2021-09-10 15:14:06,344 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .
2021-09-10 15:14:06,430 INFO
org.apache.flink.runtime.dispatcher.FileExecutionGraphInfoStore [] -
Initializing FileExecutionGraphInfoStore: Storage directory
/tmp/executionGraphStore-388e1bbd-4263-4a3b-a921-4afd2164ab27, expiration time
3600000, maximum cache size 52428800 bytes.
2021-09-10 15:14:06,534 INFO org.apache.flink.configuration.Configuration [] -
Config uses fallback configuration key 'jobmanager.rpc.address' instead of key
'rest.address'
2021-09-10 15:14:06,535 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Upload
directory /tmp/flink-web-820a0acb-c0ad-4bc0-871b-0fb546f21993/flink-web-upload
does not exist.
2021-09-10 15:14:06,536 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Created
directory /tmp/flink-web-820a0acb-c0ad-4bc0-871b-0fb546f21993/flink-web-upload
for file uploads.
2021-09-10 15:14:06,537 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Starting rest
endpoint.
2021-09-10 15:14:07,046 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of
main cluster component log file:
/opt/flink/log/flink--kubernetes-session-0-my-first-flink-clustercd-6d59756c7c-9fb7s.log
2021-09-10 15:14:07,046 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of
main cluster component stdout file:
/opt/flink/log/flink--kubernetes-session-0-my-first-flink-clustercd-6d59756c7c-9fb7s.out
2021-09-10 15:14:07,278 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Rest endpoint
listening at my-first-flink-clustercd.default:8081
2021-09-10 15:14:07,279 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] -
http://my-first-flink-clustercd.default:8081 was granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2021-09-10 15:14:07,306 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Web frontend
listening at http://my-first-flink-clustercd.default:8081.
2021-09-10 15:14:07,330 INFO
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived
from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than
its min value 192.000mb (201326592 bytes), min value will be used instead
2021-09-10 15:14:08,109 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: blob.server.port, 6124
2021-09-10 15:14:08,110 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-09-10 15:14:08,110 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2021-09-10 15:14:08,110 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.execution.failover-strategy,
region
2021-09-10 15:14:08,111 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.address,
my-first-flink-clustercd.default
2021-09-10 15:14:08,112 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.target, kubernetes-session
2021-09-10 15:14:08,113 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-09-10 15:14:08,113 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.port, 6123
2021-09-10 15:14:08,114 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.cluster-id,
my-first-flink-clustercd
2021-09-10 15:14:08,115 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.rpc.port, 6122
2021-09-10 15:14:08,116 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-09-10 15:14:08,117 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: parallelism.default, 1
2021-09-10 15:14:08,117 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-09-10 15:14:08,122 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
akka://flink/user/rpc/resourcemanager_0 .
2021-09-10 15:14:08,143 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Start SessionDispatcherLeaderProcess.
2021-09-10 15:14:08,148 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Starting the resource manager.
2021-09-10 15:14:08,208 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Recover all persisted job graphs.
2021-09-10 15:14:08,209 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Successfully recovered 0 persisted job graphs.
2021-09-10 15:14:08,245 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/rpc/dispatcher_1 .
2021-09-10 15:14:09,146 WARN
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec
Failure: HTTP 403, Status: 403 - pods is forbidden: User
"system:serviceaccount:default:default" cannot watch resource "pods" in API
group "" in the namespace "default"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_302]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
2021-09-10 15:14:09,207 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesPodsWatcher [] - The
watcher is closing.
2021-09-10 15:14:09,208 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager []
- Closing the slot manager.
2021-09-10 15:14:09,209 ERROR
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Fatal error occurred in ResourceManager.
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException:
Could not start the ResourceManager
akka.tcp://[email protected]:6123/user/rpc/resourcemanager_0
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:239)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.13.2.jar:1.13.2]
Caused by:
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException:
Cannot initialize resource provider.
at
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:156)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:251)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:235)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
... 20 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: pods is
forbidden: User "system:serviceaccount:default:default" cannot watch resource
"pods" in API group "" in the namespace "default"
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:203)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_302]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_302]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:144)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:755)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:739)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:70)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.watchPodsAndDoCallback(Fabric8FlinkKubeClient.java:227)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.KubernetesResourceManagerDriver.watchTaskManagerPods(KubernetesResourceManagerDriver.java:331)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.KubernetesResourceManagerDriver.initializeInternal(KubernetesResourceManagerDriver.java:103)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.active.AbstractResourceManagerDriver.initialize(AbstractResourceManagerDriver.java:81)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:154)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:251)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:235)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.13.2.jar:1.13.2]
2021-09-10 15:14:09,214 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred
in the cluster entrypoint.
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException:
Could not start the ResourceManager
akka.tcp://[email protected]:6123/user/rpc/resourcemanager_0
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:239)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.13.2.jar:1.13.2]
Caused by:
org.apache.flink.runtime.resourcemanager.exceptions.ResourceManagerException:
Cannot initialize resource provider.
at
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:156)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:251)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:235)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
... 20 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: pods is
forbidden: User "system:serviceaccount:default:default" cannot watch resource
"pods" in API group "" in the namespace "default"
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:203)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_302]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_302]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_302]
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:144)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:755)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:739)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:70)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.watchPodsAndDoCallback(Fabric8FlinkKubeClient.java:227)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.KubernetesResourceManagerDriver.watchTaskManagerPods(KubernetesResourceManagerDriver.java:331)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.kubernetes.KubernetesResourceManagerDriver.initializeInternal(KubernetesResourceManagerDriver.java:103)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.active.AbstractResourceManagerDriver.initialize(AbstractResourceManagerDriver.java:81)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.initialize(ActiveResourceManager.java:154)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.startResourceManagerServices(ResourceManager.java:251)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.resourcemanager.ResourceManager.onStart(ResourceManager.java:235)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
~[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.13.2.jar:1.13.2]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.13.2.jar:1.13.2]
Exception in thread "OkHttp Dispatcher"
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@393ed1
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@1fd81363[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at
java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at
java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:305)
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:50)
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:218)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-09-10 15:14:09,227 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting
KubernetesSessionClusterEntrypoint down with application status UNKNOWN.
Diagnostics Cluster entrypoint has been closed externally..
2021-09-10 15:14:09,229 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Shutting down
rest endpoint.
2021-09-10 15:14:09,229 INFO org.apache.flink.runtime.blob.BlobServer [] -
Stopped BLOB server at 0.0.0.0:6124
{code}
{code:java}
kevin@road-condition-vm-flink-client:~$ kubectl get po
my-first-flink-clustercd-6d59756c7c-9fb7s -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-09-10T00:45:02Z"
generateName: my-first-flink-clustercd-6d59756c7c-
labels:
app: my-first-flink-clustercd
component: jobmanager
pod-template-hash: 6d59756c7c
type: flink-native-kubernetes
name: my-first-flink-clustercd-6d59756c7c-9fb7s
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: my-first-flink-clustercd-6d59756c7c
uid: d500b8e0-df0d-4544-b219-b36c83a3c1a5
resourceVersion: "3569128"
uid: c4fb95a0-b4c8-4dd5-828b-7f0f9077c09f
spec:
containers:
- args:
- bash
- -c
- kubernetes-jobmanager.sh kubernetes-session
command:
- /docker-entrypoint.sh
env:
- name: _POD_IP_ADDRESS
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: apache/flink:1.13.2-scala_2.11
imagePullPolicy: IfNotPresent
name: flink-main-container
ports:
- containerPort: 8081
name: rest
protocol: TCP
- containerPort: 6123
name: jobmanager-rpc
protocol: TCP
- containerPort: 6124
name: blobserver
protocol: TCP
resources:
limits:
cpu: "1"
memory: 1600Mi
requests:
cpu: "1"
memory: 1600Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/flink/conf
name: flink-config-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5crsm
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: aks-agentpool-43117142-vmss000002
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
- configMap:
defaultMode: 420
items:
- key: logback-console.xml
path: logback-console.xml
- key: log4j-console.properties
path: log4j-console.properties
- key: flink-conf.yaml
path: flink-conf.yaml
name: flink-config-my-first-flink-clustercd
name: flink-config-volume
- name: default-token-5crsm
secret:
defaultMode: 420
secretName: default-token-5crsm
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-09-10T00:47:00Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-09-10T15:19:31Z"
message: 'containers with unready status: [flink-main-container]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-09-10T15:19:31Z"
message: 'containers with unready status: [flink-main-container]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-09-10T00:47:00Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID:
containerd://005e86da1e020dec3313103c1231bb916921545fd7b091007e7ef412d88917cd
image: docker.io/apache/flink:1.13.2-scala_2.11
imageID:
docker.io/apache/flink@sha256:2da83bf5f7437769ba1f04caed217d20bb02494a018162bfc6b2467e1914ad77
lastState:
terminated:
containerID:
containerd://005e86da1e020dec3313103c1231bb916921545fd7b091007e7ef412d88917cd
exitCode: 239
finishedAt: "2021-09-10T15:19:30Z"
reason: Error
startedAt: "2021-09-10T15:19:17Z"
name: flink-main-container
ready: false
restartCount: 168
started: false
state:
waiting:
message: back-off 5m0s restarting failed container=flink-main-container
pod=my-first-flink-clustercd-6d59756c7c-9fb7s_default(c4fb95a0-b4c8-4dd5-828b-7f0f9077c09f)
reason: CrashLoopBackOff
hostIP: 10.240.0.5
phase: Running
podIP: 10.244.2.3
podIPs:
- ip: 10.244.2.3
qosClass: Guaranteed
startTime: "2021-09-10T00:47:00Z"{code}
> 'Run kubernetes session test (default input)' failed on Azure
> -------------------------------------------------------------
>
> Key: FLINK-21647
> URL: https://issues.apache.org/jira/browse/FLINK-21647
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
> Affects Versions: 1.13.0
> Reporter: Jark Wu
> Priority: Blocker
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14236&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529&l=2247
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
