yufan wang created FLINK-21591:
----------------------------------
Summary: Flink using OSS Backend on Native K8S found some errors
Key: FLINK-21591
URL: https://issues.apache.org/jira/browse/FLINK-21591
Project: Flink
Issue Type: Bug
Reporter: yufan wang
Flink run on Native Kubernetes Application Mode
Using OSS as the backend. When the job starts, I get some error logs with
Jobmanager and Taskmanager.
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null
start command:
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=demo \
-Dkubernetes.container.image=xx/xx/job:2.0.20 \
-Dstate.backend=filesystem \
-Dstate.checkpoints.dir=oss://xx/backend/ \
-Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
-Dfs.oss.accessKeyId=xx \
-Dfs.oss.accessKeySecret=xx \
-Dhigh-availability.storageDir=oss://xx/flink-ha/ \
-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
\
local:///opt/flink/usrlib/my-flink-job.jar
morelog:
Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824
-Xms1073741824 -XX:MaxMetaspaceSize=268435456
-Dlog.file=/opt/flink/log/jobmanager.log
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
-D jobmanager.memory.off-heap.size=134217728b -D
jobmanager.memory.jvm-overhead.min=201326592b -D
jobmanager.memory.jvm-metaspace.size=268435456b -D
jobmanager.memory.heap.size=1073741824b -D
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,721 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12,
Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user:
flink
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit
Server VM - Oracle Corporation - 1.8/25.275-b01
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size:
989 MiBytes
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
/usr/local/openjdk-8
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency
available
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-XX:MaxMetaspaceSize=268435456
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog.file=/opt/flink/log/jobmanager.log
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.off-heap.size=134217728b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.min=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-metaspace.size=268435456b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.heap.size=1073741824b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
/opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX
signal handlers for [TERM, HUP, INT]
2021-03-04 02:54:50,788 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: blob.server.port, 6124
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.execution.failover-strategy,
region
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.storageDir,
oss://***/flink-ha/
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:50,790 WARN org.apache.flink.configuration.GlobalConfiguration
[] - Error while trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.container.image,
**/**/job:2.0.20
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: parallelism.default, 1
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:50,791 WARN org.apache.flink.configuration.GlobalConfiguration
[] - Error while trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.target, kubernetes-application
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.attached, true
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.endpoint,
oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: state.backend, filesystem
2021-03-04 02:54:50,794 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:51,100 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint.
2021-03-04 02:54:51,108 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default
filesystem.
2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is
not in the classpath/dependencies. The extended set of supported File Systems
via Hadoop is not available.
2021-03-04 02:54:51,188 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security
context.
2021-03-04 02:54:51,195 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-03-04 02:54:51,198 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be
created as /tmp/jaas-5821322633800809072.conf.
2021-03-04 02:54:51,202 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2021-03-04 02:54:51,203 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster
services.
2021-03-04 02:54:51,219 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start
actor system, external address 172.18.242.149:6123, bind address 0.0.0.0:6123.
2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger
started
2021-03-04 02:54:52,278 INFO akka.remote.Remoting [] - Starting remoting
2021-03-04 02:54:52,407 INFO akka.remote.Remoting [] - Remoting started;
listening on addresses :[akka.tcp://flink@172.18.242.149:6123]
2021-03-04 02:54:52,519 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started
at akka.tcp://flink@172.18.242.149:6123
2021-03-04 02:54:53,499 INFO org.apache.flink.runtime.blob.FileSystemBlobStore
[] - Creating highly available BLOB storage directory at
oss://***/flink-ha/demo/blob
2021-03-04 02:54:53,855 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null
2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer [] -
Created BLOB server storage directory
/tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer [] -
Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max
backlog: 1000
2021-03-04 02:54:53,877 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter
configured, no metrics will be exposed/reported.
2021-03-04 02:54:53,879 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start
actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger
started
2021-03-04 02:54:53,900 INFO akka.remote.Remoting [] - Starting remoting
2021-03-04 02:54:53,907 INFO akka.remote.Remoting [] - Remoting started;
listening on addresses :[akka.tcp://flink-metrics@172.18.242.149:45382]
2021-03-04 02:54:53,979 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started
at akka.tcp://flink-metrics@172.18.242.149:45382
2021-03-04 02:54:53,990 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .
2021-03-04 02:54:54,082 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
does not exist.
2021-03-04 02:54:54,083 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
for file uploads.
2021-03-04 02:54:54,094 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting
rest endpoint.
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of
main cluster component log file: /opt/flink/log/jobmanager.log
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of
main cluster component stdout file: /opt/flink/log/jobmanager.out
2021-03-04 02:54:54,605 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest
endpoint listening at 172.18.242.149:8081
2021-03-04 02:54:54,788 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-restserver-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:54,982 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver\{configMapName='demo-restserver-leader'}.
2021-03-04 02:54:54,983 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend
listening at http://172.18.242.149:8081.
2021-03-04 02:54:54,993 INFO
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived
from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than
its min value 192.000mb (201326592 bytes), min value will be used instead
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: blob.server.port, 6124
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.execution.failover-strategy,
region
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.storageDir,
oss://***/flink-ha/
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:55,006 WARN org.apache.flink.configuration.GlobalConfiguration
[] - Error while trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: kubernetes.container.image,
**/**/job:2.0.20
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: parallelism.default, 1
2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:55,073 WARN org.apache.flink.configuration.GlobalConfiguration
[] - Error while trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.target, kubernetes-application
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.attached, true
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: fs.oss.endpoint,
oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: state.backend, filesystem
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration
[] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:55,081 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
akka://flink/user/rpc/resourcemanager_0 .
2021-03-04 02:54:55,093 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-dispatcher-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:55,099 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver\{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,279 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver\{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:55,285 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver\{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,492 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-restserver-leader.
2021-03-04 02:54:55,493 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
http://172.18.242.149:8081 was granted leadership with
leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
2021-03-04 02:54:55,494 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-dispatcher-leader.
2021-03-04 02:54:55,578 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Start SessionDispatcherLeaderProcess.
2021-03-04 02:54:55,581 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Recover all persisted job graphs.
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job ids
[] from KubernetesStateHandleStore\{configMapName='demo-dispatcher-leader'}
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Successfully recovered 0 persisted job graphs.
2021-03-04 02:54:55,595 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/rpc/dispatcher_1 .
2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils [] - Starting
program (detached: false)
2021-03-04 02:54:56,089 INFO
org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0
pods from previous attempts, current attempt id is 1.
2021-03-04 02:54:56,090 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Recovered 0 workers from previous attempt.
2021-03-04 02:54:56,090 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:56,097 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver\{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:56,175 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-resourcemanager-leader.
2021-03-04 02:54:56,274 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
ResourceManager akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0
was granted leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
2021-03-04 02:54:56,278 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
Starting the SlotManager.
2021-03-04 02:54:56,512 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it cannot be
used as a POJO type and must be processed as GenericType. Please read the Flink
documentation on "Data Types & Serialization" for details of the effect on
performance.
2021-03-04 02:54:56,519 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it cannot be
used as a POJO type and must be processed as GenericType. Please read the Flink
documentation on "Data Types & Serialization" for details of the effect on
performance.
2021-03-04 02:54:56,575 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionKeyBy is missing a default constructor so
it cannot be used as a POJO type and must be processed as GenericType. Please
read the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,614 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionSum is missing a default constructor so it
cannot be used as a POJO type and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Job 00000000000000000000000000000000 is submitted.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Submitting Job with JobId=00000000000000000000000000000000.
2021-03-04 02:54:57,913 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C38389D6321
[HostId]: null
2021-03-04 02:54:57,915 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C3838AA6321
[HostId]: null
2021-03-04 02:54:58,902 WARN
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled
exception
java.io.IOException: Connection reset by peer // Exception ignored, caused by
cloud provider LoadBalance
...
2021-03-04 02:55:02,535 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838DC9C21
[HostId]: null
2021-03-04 02:55:02,537 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E69C21
[HostId]: null
2021-03-04 02:55:02,634 WARN
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to find
'mime.types' file in classpath
2021-03-04 02:55:02,768 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C38387B9F21
[HostId]: null
2021-03-04 02:55:02,770 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838869F21
[HostId]: null
2021-03-04 02:55:02,888 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E4A021
[HostId]: null
2021-03-04 02:55:02,890 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838ECA021
[HostId]: null
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
