[
https://issues.apache.org/jira/browse/FLINK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yufan wang updated FLINK-21591:
-------------------------------
Description:
Flink run on Native Kubernetes Application Mode
Using OSS as the backend. When the job starts, I get some error logs with
Jobmanager and Taskmanager.
{code:java}
2021-03-04 02:55:02,634 WARN
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
find 'mime.types' file in classpath
2021-03-04 02:55:02,768 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C38387B9F21
[HostId]: null
{code}
start command:
{code:java}
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=demo \
-Dkubernetes.container.image=xx/xx/job:2.0.20 \
-Dstate.backend=filesystem \
-Dstate.checkpoints.dir=oss://xx/backend/ \
-Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
-Dfs.oss.accessKeyId=xx \
-Dfs.oss.accessKeySecret=xx \
-Dhigh-availability.storageDir=oss://xx/flink-ha/ \
-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
\
local:///opt/flink/usrlib/my-flink-job.jar
{code}
morelog:
{code:java}
Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824
-Xms1073741824 -XX:MaxMetaspaceSize=268435456
-Dlog.file=/opt/flink/log/jobmanager.log
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
-D jobmanager.memory.off-heap.size=134217728b -D
jobmanager.memory.jvm-overhead.min=201326592b -D
jobmanager.memory.jvm-metaspace.size=268435456b -D
jobmanager.memory.heap.size=1073741824b -D
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,721 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12,
Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current
user: flink
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap
size: 989 MiBytes
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
/usr/local/openjdk-8
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop
Dependency available
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xmx1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xms1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-XX:MaxMetaspaceSize=268435456
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog.file=/opt/flink/log/jobmanager.log
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program
Arguments:
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.off-heap.size=134217728b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.min=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-metaspace.size=268435456b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.heap.size=1073741824b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
/opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered
UNIX signal handlers for [TERM, HUP, INT]
2021-03-04 02:54:50,788 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:50,790 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, 1
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:50,791 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-application
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.attached, true
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.backend, filesystem
2021-03-04 02:54:50,794 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:51,100 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint.
2021-03-04 02:54:51,108 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
default filesystem.
2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem
[] - Hadoop is not in the classpath/dependencies. The extended set
of supported File Systems via Hadoop is not available.
2021-03-04 02:54:51,188 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
security context.
2021-03-04 02:54:51,195 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-03-04 02:54:51,198 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
will be created as /tmp/jaas-5821322633800809072.conf.
2021-03-04 02:54:51,202 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2021-03-04 02:54:51,203 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing
cluster services.
2021-03-04 02:54:51,219 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address 172.18.242.149:6123, bind address
0.0.0.0:6123.
2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2021-03-04 02:54:52,278 INFO akka.remote.Remoting
[] - Starting remoting
2021-03-04 02:54:52,407 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink@172.18.242.149:6123]
2021-03-04 02:54:52,519 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink@172.18.242.149:6123
2021-03-04 02:54:53,499 INFO org.apache.flink.runtime.blob.FileSystemBlobStore
[] - Creating highly available BLOB storage directory at
oss://***/flink-ha/demo/blob
2021-03-04 02:54:53,855 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null
2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer
[] - Created BLOB server storage directory
/tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer
[] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests:
50 - max backlog: 1000
2021-03-04 02:54:53,877 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics
reporter configured, no metrics will be exposed/reported.
2021-03-04 02:54:53,879 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2021-03-04 02:54:53,900 INFO akka.remote.Remoting
[] - Starting remoting
2021-03-04 02:54:53,907 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink-metrics@172.18.242.149:45382]
2021-03-04 02:54:53,979 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink-metrics@172.18.242.149:45382
2021-03-04 02:54:53,990 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .
2021-03-04 02:54:54,082 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
does not exist.
2021-03-04 02:54:54,083 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
for file uploads.
2021-03-04 02:54:54,094 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting
rest endpoint.
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component log file: /opt/flink/log/jobmanager.log
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component stdout file: /opt/flink/log/jobmanager.out
2021-03-04 02:54:54,605 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest
endpoint listening at 172.18.242.149:8081
2021-03-04 02:54:54,788 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-restserver-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:54,982 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-restserver-leader'}.
2021-03-04 02:54:54,983 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend
listening at http://172.18.242.149:8081.
2021-03-04 02:54:54,993 INFO
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived
from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than
its min value 192.000mb (201326592 bytes), min value will be used instead
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:55,006 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, 1
2021-03-04 02:54:55,073 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:55,073 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:55,073 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-application
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.attached, true
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.backend, filesystem
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:55,081 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
akka://flink/user/rpc/resourcemanager_0 .
2021-03-04 02:54:55,093 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-dispatcher-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:55,099 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,279 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:55,285 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,492 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-restserver-leader.
2021-03-04 02:54:55,493 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
http://172.18.242.149:8081 was granted leadership with
leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
2021-03-04 02:54:55,494 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-dispatcher-leader.
2021-03-04 02:54:55,578 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Start SessionDispatcherLeaderProcess.
2021-03-04 02:54:55,581 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Recover all persisted job graphs.
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job
ids [] from KubernetesStateHandleStore{configMapName='demo-dispatcher-leader'}
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Successfully recovered 0 persisted job graphs.
2021-03-04 02:54:55,595 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/rpc/dispatcher_1 .
2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils
[] - Starting program (detached: false)
2021-03-04 02:54:56,089 INFO
org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0
pods from previous attempts, current attempt id is 1.
2021-03-04 02:54:56,090 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Recovered 0 workers from previous attempt.
2021-03-04 02:54:56,090 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:56,097 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:56,175 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-resourcemanager-leader.
2021-03-04 02:54:56,274 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
ResourceManager akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0
was granted leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
2021-03-04 02:54:56,278 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
Starting the SlotManager.
2021-03-04 02:54:56,512 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it
cannot be used as a POJO type and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,519 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it
cannot be used as a POJO type and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,575 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionKeyBy is missing a default
constructor so it cannot be used as a POJO type and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-03-04 02:54:56,614 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionSum is missing a default
constructor so it cannot be used as a POJO type and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Job 00000000000000000000000000000000 is submitted.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Submitting Job with JobId=00000000000000000000000000000000.
2021-03-04 02:54:57,913 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C38389D6321
[HostId]: null
2021-03-04 02:54:57,915 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C3838AA6321
[HostId]: null
2021-03-04 02:54:58,902 WARN
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled
exception
java.io.IOException: Connection reset by peer // Ignore these exceptions,
caused by cloud provider LoadBalance
...
2021-03-04 02:55:02,535 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838DC9C21
[HostId]: null
2021-03-04 02:55:02,537 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E69C21
[HostId]: null
2021-03-04 02:55:02,634 WARN
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
find 'mime.types' file in classpath
2021-03-04 02:55:02,768 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C38387B9F21
[HostId]: null
2021-03-04 02:55:02,770 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838869F21
[HostId]: null
2021-03-04 02:55:02,888 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E4A021
[HostId]: null
2021-03-04 02:55:02,890 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838ECA021
[HostId]: null
{code}
was:
Flink run on Native Kubernetes Application Mode
Using OSS as the backend. When the job starts, I get some error logs with
Jobmanager and Taskmanager.
{code:java}
2021-03-04 02:55:02,634 WARN
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to find
'mime.types' file in classpath 2021-03-04 02:55:02,768 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to
execute HTTP request: Not Found [ErrorCode]: NoSuchKey [RequestId]:
60404C06BD813C38387B9F21 [HostId]: null
{code}
start command:
{code:java}
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=demo \
-Dkubernetes.container.image=xx/xx/job:2.0.20 \
-Dstate.backend=filesystem \
-Dstate.checkpoints.dir=oss://xx/backend/ \
-Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
-Dfs.oss.accessKeyId=xx \
-Dfs.oss.accessKeySecret=xx \
-Dhigh-availability.storageDir=oss://xx/flink-ha/ \
-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
\
local:///opt/flink/usrlib/my-flink-job.jar
{code}
morelog:
{code:java}
Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824
-Xms1073741824 -XX:MaxMetaspaceSize=268435456
-Dlog.file=/opt/flink/log/jobmanager.log
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
-D jobmanager.memory.off-heap.size=134217728b -D
jobmanager.memory.jvm-overhead.min=201326592b -D
jobmanager.memory.jvm-metaspace.size=268435456b -D
jobmanager.memory.heap.size=1073741824b -D
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,721 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12,
Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
2021-03-04 02:54:50,723 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current
user: flink
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap
size: 989 MiBytes
2021-03-04 02:54:50,724 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
/usr/local/openjdk-8
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop
Dependency available
2021-03-04 02:54:50,773 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xmx1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xms1073741824
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-XX:MaxMetaspaceSize=268435456
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog.file=/opt/flink/log/jobmanager.log
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program
Arguments:
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,775 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.off-heap.size=134217728b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.min=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-metaspace.size=268435456b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.heap.size=1073741824b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,776 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
/opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2021-03-04 02:54:50,777 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered
UNIX signal handlers for [TERM, HUP, INT]
2021-03-04 02:54:50,788 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:50,789 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:50,790 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:50,790 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, 1
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:50,791 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:50,791 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-application
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:50,792 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.attached, true
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:50,793 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.backend, filesystem
2021-03-04 02:54:50,794 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:51,100 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesApplicationClusterEntrypoint.
2021-03-04 02:54:51,108 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
default filesystem.
2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem
[] - Hadoop is not in the classpath/dependencies. The extended set
of supported File Systems via Hadoop is not available.
2021-03-04 02:54:51,188 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
security context.
2021-03-04 02:54:51,195 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-03-04 02:54:51,198 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
will be created as /tmp/jaas-5821322633800809072.conf.
2021-03-04 02:54:51,202 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2021-03-04 02:54:51,203 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing
cluster services.
2021-03-04 02:54:51,219 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address 172.18.242.149:6123, bind address
0.0.0.0:6123.
2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2021-03-04 02:54:52,278 INFO akka.remote.Remoting
[] - Starting remoting
2021-03-04 02:54:52,407 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink@172.18.242.149:6123]
2021-03-04 02:54:52,519 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink@172.18.242.149:6123
2021-03-04 02:54:53,499 INFO org.apache.flink.runtime.blob.FileSystemBlobStore
[] - Creating highly available BLOB storage directory at
oss://***/flink-ha/demo/blob
2021-03-04 02:54:53,855 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null
2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer
[] - Created BLOB server storage directory
/tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer
[] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests:
50 - max backlog: 1000
2021-03-04 02:54:53,877 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics
reporter configured, no metrics will be exposed/reported.
2021-03-04 02:54:53,879 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2021-03-04 02:54:53,900 INFO akka.remote.Remoting
[] - Starting remoting
2021-03-04 02:54:53,907 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink-metrics@172.18.242.149:45382]
2021-03-04 02:54:53,979 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink-metrics@172.18.242.149:45382
2021-03-04 02:54:53,990 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .
2021-03-04 02:54:54,082 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
does not exist.
2021-03-04 02:54:54,083 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created
directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload
for file uploads.
2021-03-04 02:54:54,094 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting
rest endpoint.
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component log file: /opt/flink/log/jobmanager.log
2021-03-04 02:54:54,412 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component stdout file: /opt/flink/log/jobmanager.out
2021-03-04 02:54:54,605 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest
endpoint listening at 172.18.242.149:8081
2021-03-04 02:54:54,788 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-restserver-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:54,982 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-restserver-leader'}.
2021-03-04 02:54:54,983 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend
listening at http://172.18.242.149:8081.
2021-03-04 02:54:54,993 INFO
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived
from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than
its min value 192.000mb (201326592 bytes), min value will be used instead
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:55,005 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:55,006 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:55,006 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, 1
2021-03-04 02:54:55,073 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:55,073 WARN
org.apache.flink.configuration.GlobalConfiguration [] - Error while
trying to split key and value in configuration file
/opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:55,073 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-application
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.attached, true
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: high-availability,
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:55,074 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: pipeline.jars,
local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.backend, filesystem
2021-03-04 02:54:55,075 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:55,081 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
akka://flink/user/rpc/resourcemanager_0 .
2021-03-04 02:54:55,093 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-dispatcher-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:55,099 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,279 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:55,285 INFO
org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
Starting DefaultLeaderRetrievalService with
KubernetesLeaderRetrievalDriver{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,492 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-restserver-leader.
2021-03-04 02:54:55,493 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
http://172.18.242.149:8081 was granted leadership with
leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
2021-03-04 02:54:55,494 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-dispatcher-leader.
2021-03-04 02:54:55,578 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Start SessionDispatcherLeaderProcess.
2021-03-04 02:54:55,581 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Recover all persisted job graphs.
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job
ids [] from KubernetesStateHandleStore{configMapName='demo-dispatcher-leader'}
2021-03-04 02:54:55,588 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Successfully recovered 0 persisted job graphs.
2021-03-04 02:54:55,595 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/rpc/dispatcher_1 .
2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils
[] - Starting program (detached: false)
2021-03-04 02:54:56,089 INFO
org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0
pods from previous attempts, current attempt id is 1.
2021-03-04 02:54:56,090 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Recovered 0 workers from previous attempt.
2021-03-04 02:54:56,090 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity
ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:56,097 INFO
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Starting DefaultLeaderElectionService with
KubernetesLeaderElectionDriver{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:56,175 INFO
org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
demo-resourcemanager-leader.
2021-03-04 02:54:56,274 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
ResourceManager akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0
was granted leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
2021-03-04 02:54:56,278 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
Starting the SlotManager.
2021-03-04 02:54:56,512 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it
cannot be used as a POJO type and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,519 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - Class com.**.entity.TransactionBase$Source is not public so it
cannot be used as a POJO type and must be processed as GenericType. Please read
the Flink documentation on "Data Types & Serialization" for details of the
effect on performance.
2021-03-04 02:54:56,575 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionKeyBy is missing a default
constructor so it cannot be used as a POJO type and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-03-04 02:54:56,614 INFO org.apache.flink.api.java.typeutils.TypeExtractor
[] - class com.**.entity.TransactionSum is missing a default
constructor so it cannot be used as a POJO type and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Job 00000000000000000000000000000000 is submitted.
2021-03-04 02:54:56,806 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Submitting Job with JobId=00000000000000000000000000000000.
2021-03-04 02:54:57,913 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C38389D6321
[HostId]: null
2021-03-04 02:54:57,915 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C3838AA6321
[HostId]: null
2021-03-04 02:54:58,902 WARN
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled
exception
java.io.IOException: Connection reset by peer // Ignore these exceptions,
caused by cloud provider LoadBalance
...
2021-03-04 02:55:02,535 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838DC9C21
[HostId]: null
2021-03-04 02:55:02,537 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E69C21
[HostId]: null
2021-03-04 02:55:02,634 WARN
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
find 'mime.types' file in classpath
2021-03-04 02:55:02,768 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C38387B9F21
[HostId]: null
2021-03-04 02:55:02,770 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838869F21
[HostId]: null
2021-03-04 02:55:02,888 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E4A021
[HostId]: null
2021-03-04 02:55:02,890 INFO
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838ECA021
[HostId]: null
{code}
> Flink using OSS Backend on Native K8S found some errors
> -------------------------------------------------------
>
> Key: FLINK-21591
> URL: https://issues.apache.org/jira/browse/FLINK-21591
> Project: Flink
> Issue Type: Bug
> Reporter: yufan wang
> Priority: Blocker
>
> Flink run on Native Kubernetes Application Mode
> Using OSS as the backend. When the job starts, I get some error logs with
> Jobmanager and Taskmanager.
> {code:java}
> 2021-03-04 02:55:02,634 WARN
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
> find 'mime.types' file in classpath
> 2021-03-04 02:55:02,768 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C38387B9F21
> [HostId]: null
> {code}
> start command:
> {code:java}
> ./bin/flink run-application \
> --target kubernetes-application \
> -Dkubernetes.cluster-id=demo \
> -Dkubernetes.container.image=xx/xx/job:2.0.20 \
> -Dstate.backend=filesystem \
> -Dstate.checkpoints.dir=oss://xx/backend/ \
> -Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
> -Dfs.oss.accessKeyId=xx \
> -Dfs.oss.accessKeySecret=xx \
> -Dhigh-availability.storageDir=oss://xx/flink-ha/ \
>
> -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> \
> local:///opt/flink/usrlib/my-flink-job.jar
> {code}
> morelog:
> {code:java}
> Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824
> -Xms1073741824 -XX:MaxMetaspaceSize=268435456
> -Dlog.file=/opt/flink/log/jobmanager.log
> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
> org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> -D jobmanager.memory.off-heap.size=134217728b -D
> jobmanager.memory.jvm-overhead.min=201326592b -D
> jobmanager.memory.jvm-metaspace.size=268435456b -D
> jobmanager.memory.heap.size=1073741824b -D
> jobmanager.memory.jvm-overhead.max=201326592b
> 2021-03-04 02:54:50,721 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> --------------------------------------------------------------------------------
> 2021-03-04 02:54:50,723 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
> KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12,
> Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
> 2021-03-04 02:54:50,723 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current
> user: flink
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
> Hadoop/Kerberos user: <no hadoop dependency found>
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM:
> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum
> heap size: 989 MiBytes
> 2021-03-04 02:54:50,724 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
> /usr/local/openjdk-8
> 2021-03-04 02:54:50,773 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop
> Dependency available
> 2021-03-04 02:54:50,773 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM
> Options:
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Xmx1073741824
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Xms1073741824
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -XX:MaxMetaspaceSize=268435456
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Dlog.file=/opt/flink/log/jobmanager.log
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
> 2021-03-04 02:54:50,774 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program
> Arguments:
> 2021-03-04 02:54:50,775 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,775 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> jobmanager.memory.off-heap.size=134217728b
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> jobmanager.memory.jvm-overhead.min=201326592b
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> jobmanager.memory.jvm-metaspace.size=268435456b
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> jobmanager.memory.heap.size=1073741824b
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> jobmanager.memory.jvm-overhead.max=201326592b
> 2021-03-04 02:54:50,776 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
> /opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
> 2021-03-04 02:54:50,777 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> --------------------------------------------------------------------------------
> 2021-03-04 02:54:50,777 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered
> UNIX signal handlers for [TERM, HUP, INT]
> 2021-03-04 02:54:50,788 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: blob.server.port, 6124
> 2021-03-04 02:54:50,789 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.execution.failover-strategy, region
> 2021-03-04 02:54:50,789 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.cluster-id, demo
> 2021-03-04 02:54:50,789 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2021-03-04 02:54:50,789 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.cluster-id, demo
> 2021-03-04 02:54:50,790 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.accessKeySecret, ******
> 2021-03-04 02:54:50,790 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.accessKeyId, ***
> 2021-03-04 02:54:50,790 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.storageDir, oss://***/flink-ha/
> 2021-03-04 02:54:50,790 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-03-04 02:54:50,790 WARN
> org.apache.flink.configuration.GlobalConfiguration [] - Error while
> trying to split key and value in configuration file
> /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
> 2021-03-04 02:54:50,791 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.container.image, **/**/job:2.0.20
> 2021-03-04 02:54:50,791 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: parallelism.default, 1
> 2021-03-04 02:54:50,791 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-03-04 02:54:50,791 WARN
> org.apache.flink.configuration.GlobalConfiguration [] - Error while
> trying to split key and value in configuration file
> /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
> 2021-03-04 02:54:50,791 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.jobmanager.port, 6123
> 2021-03-04 02:54:50,791 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.memory.process.size, 1728m
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.internal.jobmanager.entrypoint.class,
> org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.target, kubernetes-application
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.memory.process.size, 1600m
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.rpc.port, 6123
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.rpc.port, 6122
> 2021-03-04 02:54:50,792 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.attached, true
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: internal.cluster.execution-mode, NORMAL
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability,
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.shutdown-on-attached-exit, false
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: pipeline.jars,
> local:///opt/flink/usrlib/my-flink-job.jar
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
> 2021-03-04 02:54:50,793 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.backend, filesystem
> 2021-03-04 02:54:50,794 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.checkpoints.dir, oss://***/backend/
> 2021-03-04 02:54:51,100 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
> KubernetesApplicationClusterEntrypoint.
> 2021-03-04 02:54:51,108 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
> default filesystem.
> 2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem
> [] - Hadoop is not in the classpath/dependencies. The extended
> set of supported File Systems via Hadoop is not available.
> 2021-03-04 02:54:51,188 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
> security context.
> 2021-03-04 02:54:51,195 INFO
> org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
> create Hadoop Security Module because Hadoop cannot be found in the Classpath.
> 2021-03-04 02:54:51,198 INFO
> org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
> will be created as /tmp/jaas-5821322633800809072.conf.
> 2021-03-04 02:54:51,202 INFO
> org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
> Cannot install HadoopSecurityContext because Hadoop cannot be found in the
> Classpath.
> 2021-03-04 02:54:51,203 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
> Initializing cluster services.
> 2021-03-04 02:54:51,219 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
> start actor system, external address 172.18.242.149:6123, bind address
> 0.0.0.0:6123.
> 2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger
> [] - Slf4jLogger started
> 2021-03-04 02:54:52,278 INFO akka.remote.Remoting
> [] - Starting remoting
> 2021-03-04 02:54:52,407 INFO akka.remote.Remoting
> [] - Remoting started; listening on addresses
> :[akka.tcp://flink@172.18.242.149:6123]
> 2021-03-04 02:54:52,519 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor
> system started at akka.tcp://flink@172.18.242.149:6123
> 2021-03-04 02:54:53,499 INFO
> org.apache.flink.runtime.blob.FileSystemBlobStore [] - Creating
> highly available BLOB storage directory at oss://***/flink-ha/demo/blob
> 2021-03-04 02:54:53,855 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404BFDBD813C3838043521
> [HostId]: null
> 2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Created BLOB server storage directory
> /tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
> 2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Started BLOB server at 0.0.0.0:6124 - max concurrent
> requests: 50 - max backlog: 1000
> 2021-03-04 02:54:53,877 INFO
> org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics
> reporter configured, no metrics will be exposed/reported.
> 2021-03-04 02:54:53,879 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
> start actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
> 2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger
> [] - Slf4jLogger started
> 2021-03-04 02:54:53,900 INFO akka.remote.Remoting
> [] - Starting remoting
> 2021-03-04 02:54:53,907 INFO akka.remote.Remoting
> [] - Remoting started; listening on addresses
> :[akka.tcp://flink-metrics@172.18.242.149:45382]
> 2021-03-04 02:54:53,979 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor
> system started at akka.tcp://flink-metrics@172.18.242.149:45382
> 2021-03-04 02:54:53,990 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting
> RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at
> akka://flink-metrics/user/rpc/MetricQueryService .
> 2021-03-04 02:54:54,082 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload
> directory
> /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload does not
> exist.
> 2021-03-04 02:54:54,083 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created
> directory
> /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload for file
> uploads.
> 2021-03-04 02:54:54,094 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting
> rest endpoint.
> 2021-03-04 02:54:54,412 INFO
> org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
> location of main cluster component log file: /opt/flink/log/jobmanager.log
> 2021-03-04 02:54:54,412 INFO
> org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
> location of main cluster component stdout file: /opt/flink/log/jobmanager.out
> 2021-03-04 02:54:54,605 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest
> endpoint listening at 172.18.242.149:8081
> 2021-03-04 02:54:54,788 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> Create KubernetesLeaderElector demo-restserver-leader with lock identity
> ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:54,982 INFO
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Starting DefaultLeaderElectionService with
> KubernetesLeaderElectionDriver{configMapName='demo-restserver-leader'}.
> 2021-03-04 02:54:54,983 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web
> frontend listening at http://172.18.242.149:8081.
> 2021-03-04 02:54:54,993 INFO
> org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The
> derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is
> less than its min value 192.000mb (201326592 bytes), min value will be used
> instead
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: blob.server.port, 6124
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.execution.failover-strategy, region
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.cluster-id, demo
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.cluster-id, demo
> 2021-03-04 02:54:55,005 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.accessKeySecret, ******
> 2021-03-04 02:54:55,006 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.accessKeyId, ***
> 2021-03-04 02:54:55,006 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.storageDir, oss://***/flink-ha/
> 2021-03-04 02:54:55,006 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-03-04 02:54:55,006 WARN
> org.apache.flink.configuration.GlobalConfiguration [] - Error while
> trying to split key and value in configuration file
> /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
> 2021-03-04 02:54:55,006 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.container.image, **/**/job:2.0.20
> 2021-03-04 02:54:55,006 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: parallelism.default, 1
> 2021-03-04 02:54:55,073 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-03-04 02:54:55,073 WARN
> org.apache.flink.configuration.GlobalConfiguration [] - Error while
> trying to split key and value in configuration file
> /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
> 2021-03-04 02:54:55,073 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability.jobmanager.port, 6123
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.memory.process.size, 1728m
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: kubernetes.internal.jobmanager.entrypoint.class,
> org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.target, kubernetes-application
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.memory.process.size, 1600m
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.rpc.port, 6123
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.rpc.port, 6122
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.attached, true
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: internal.cluster.execution-mode, NORMAL
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: high-availability,
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> 2021-03-04 02:54:55,074 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: execution.shutdown-on-attached-exit, false
> 2021-03-04 02:54:55,075 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: pipeline.jars,
> local:///opt/flink/usrlib/my-flink-job.jar
> 2021-03-04 02:54:55,075 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
> 2021-03-04 02:54:55,075 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.backend, filesystem
> 2021-03-04 02:54:55,075 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: state.checkpoints.dir, oss://***/backend/
> 2021-03-04 02:54:55,081 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting
> RPC endpoint for
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
> akka://flink/user/rpc/resourcemanager_0 .
> 2021-03-04 02:54:55,093 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> Create KubernetesLeaderElector demo-dispatcher-leader with lock identity
> ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:55,099 INFO
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Starting DefaultLeaderElectionService with
> KubernetesLeaderElectionDriver{configMapName='demo-dispatcher-leader'}.
> 2021-03-04 02:54:55,279 INFO
> org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
> Starting DefaultLeaderRetrievalService with
> KubernetesLeaderRetrievalDriver{configMapName='demo-resourcemanager-leader'}.
> 2021-03-04 02:54:55,285 INFO
> org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] -
> Starting DefaultLeaderRetrievalService with
> KubernetesLeaderRetrievalDriver{configMapName='demo-dispatcher-leader'}.
> 2021-03-04 02:54:55,492 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
> demo-restserver-leader.
> 2021-03-04 02:54:55,493 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> http://172.18.242.149:8081 was granted leadership with
> leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
> 2021-03-04 02:54:55,494 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
> demo-dispatcher-leader.
> 2021-03-04 02:54:55,578 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess []
> - Start SessionDispatcherLeaderProcess.
> 2021-03-04 02:54:55,581 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess []
> - Recover all persisted job graphs.
> 2021-03-04 02:54:55,588 INFO
> org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved
> job ids [] from
> KubernetesStateHandleStore{configMapName='demo-dispatcher-leader'}
> 2021-03-04 02:54:55,588 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess []
> - Successfully recovered 0 persisted job graphs.
> 2021-03-04 02:54:55,595 INFO
> org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting
> RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
> akka://flink/user/rpc/dispatcher_1 .
> 2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils
> [] - Starting program (detached: false)
> 2021-03-04 02:54:56,089 INFO
> org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0
> pods from previous attempts, current attempt id is 1.
> 2021-03-04 02:54:56,090 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Recovered 0 workers from previous attempt.
> 2021-03-04 02:54:56,090 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity
> ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:56,097 INFO
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Starting DefaultLeaderElectionService with
> KubernetesLeaderElectionDriver{configMapName='demo-resourcemanager-leader'}.
> 2021-03-04 02:54:56,175 INFO
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] -
> New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for
> demo-resourcemanager-leader.
> 2021-03-04 02:54:56,274 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> ResourceManager
> akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0 was granted
> leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
> 2021-03-04 02:54:56,278 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
> Starting the SlotManager.
> 2021-03-04 02:54:56,512 INFO
> org.apache.flink.api.java.typeutils.TypeExtractor [] - Class
> com.**.entity.TransactionBase$Source is not public so it cannot be used as a
> POJO type and must be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance.
> 2021-03-04 02:54:56,519 INFO
> org.apache.flink.api.java.typeutils.TypeExtractor [] - Class
> com.**.entity.TransactionBase$Source is not public so it cannot be used as a
> POJO type and must be processed as GenericType. Please read the Flink
> documentation on "Data Types & Serialization" for details of the effect on
> performance.
> 2021-03-04 02:54:56,575 INFO
> org.apache.flink.api.java.typeutils.TypeExtractor [] - class
> com.**.entity.TransactionKeyBy is missing a default constructor so it cannot
> be used as a POJO type and must be processed as GenericType. Please read the
> Flink documentation on "Data Types & Serialization" for details of the effect
> on performance.
> 2021-03-04 02:54:56,614 INFO
> org.apache.flink.api.java.typeutils.TypeExtractor [] - class
> com.**.entity.TransactionSum is missing a default constructor so it cannot be
> used as a POJO type and must be processed as GenericType. Please read the
> Flink documentation on "Data Types & Serialization" for details of the effect
> on performance.
> 2021-03-04 02:54:56,806 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Job 00000000000000000000000000000000 is submitted.
> 2021-03-04 02:54:56,806 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor []
> - Submitting Job with JobId=00000000000000000000000000000000.
> 2021-03-04 02:54:57,913 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C01BD813C38389D6321
> [HostId]: null
> 2021-03-04 02:54:57,915 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C01BD813C3838AA6321
> [HostId]: null
> 2021-03-04 02:54:58,902 WARN
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled
> exception
> java.io.IOException: Connection reset by peer // Ignore these exceptions,
> caused by cloud provider LoadBalance
> ...
> 2021-03-04 02:55:02,535 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838DC9C21
> [HostId]: null
> 2021-03-04 02:55:02,537 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838E69C21
> [HostId]: null
> 2021-03-04 02:55:02,634 WARN
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to
> find 'mime.types' file in classpath
> 2021-03-04 02:55:02,768 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C38387B9F21
> [HostId]: null
> 2021-03-04 02:55:02,770 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838869F21
> [HostId]: null
> 2021-03-04 02:55:02,888 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838E4A021
> [HostId]: null
> 2021-03-04 02:55:02,890 INFO
> org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] -
> [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838ECA021
> [HostId]: null
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
