附件为错误日志。哪位大佬帮忙分析下。
2020-06-20 08:39:47,829 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
--------------------------------------------------------------------------------
2020-06-20 08:39:47,830 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Starting YARN TaskExecutor runner (Version: 1.9.2, Rev:c9d2c90,
Date:24.01.2020 @ 08:44:30 CST)
2020-06-20 08:39:47,830 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- OS current user: ocdc
2020-06-20 08:39:48,235 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current Hadoop/Kerberos user: ocdp
2020-06-20 08:39:48,235 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation -
1.8/25.152-b16
2020-06-20 08:39:48,235 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Maximum heap size: 5300 MiBytes
2020-06-20 08:39:48,235 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JAVA_HOME: /usr/local/jdk1.8.0_152
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Hadoop version: 2.7.3.2.6.0.3-8
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM Options:
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xms5529m
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xmx5529m
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -XX:MaxDirectMemorySize=2663m
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dfile.encoding=UTF-8
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
-Dlog.file=/data09/hadoop/yarn/log/application_1567067657620_0251/container_e07_1567067657620_0251_01_000005/taskmanager.log
2020-06-20 08:39:48,236 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlogback.configurationFile=file:./logback.xml
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlog4j.configuration=file:./log4j.properties
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Program Arguments:
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- --configDir
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- .
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dweb.port=0
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Djobmanager.rpc.address=audit-dp04
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dtaskmanager.memory.size=4058744064b
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
-Dweb.tmpdir=/tmp/flink-web-3ead1dd7-b12c-4a61-9a5d-793743c58302
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Djobmanager.rpc.port=57053
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Drest.address=audit-dp04
2020-06-20 08:39:48,237 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Classpath:
lib/flink-table-blink_2.12-1.9.2.jar:lib/flink-table_2.12-1.9.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.0.3-8-tests.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/joda-time-2.9.4.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.7.0.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.0.3-8.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.7.0.2.6.0.3-8.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.0.3-8-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.0.3-8-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.0.3-8.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0.jar
2020-06-20 08:39:48,238 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
--------------------------------------------------------------------------------
2020-06-20 08:39:48,239 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Registered UNIX signal handlers for [TERM, HUP, INT]
2020-06-20 08:39:48,242 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current working Directory:
/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/container_e07_1567067657620_0251_01_000005
2020-06-20 08:39:48,251 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.checkpoints.num, 6
2020-06-20 08:39:48,251 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.execution.failover-strategy, region
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: high-availability.cluster-id,
application_1567067657620_0251
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: env.java.opts.taskmanager, "-Dfile.encoding=UTF-8"
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2020-06-20 08:39:48,252 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 8192m
2020-06-20 08:39:48,253 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 3096m
2020-06-20 08:39:48,301 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current working/local Directory:
/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251
2020-06-20 08:39:48,303 INFO
org.apache.flink.runtime.clusterframework.BootstrapTools - Setting
directories for temporary files to:
/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251,/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251
2020-06-20 08:39:48,303 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- TM: remote keytab path obtained null
2020-06-20 08:39:48,303 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- TM: remote keytab principal obtained null
2020-06-20 08:39:48,304 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- YARN daemon is running as: ocdp Yarn client user obtainer: ocdp
2020-06-20 08:39:48,347 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set
to ocdp (auth:SIMPLE)
2020-06-20 08:39:48,783 WARN org.apache.flink.configuration.Configuration
- Config uses deprecated configuration key 'web.port' instead of
proper key 'rest.port'
2020-06-20 08:39:48,786 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Using
configured hostname/address for TaskManager: audit-dp03.
2020-06-20 08:39:48,787 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start
actor system at audit-dp03:0
2020-06-20 08:39:49,381 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2020-06-20 08:39:49,410 INFO akka.remote.Remoting
- Starting remoting
2020-06-20 08:39:49,552 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@audit-dp03:35130]
2020-06-20 08:39:49,771 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system
started at akka.tcp://flink@audit-dp03:35130
2020-06-20 08:39:49,786 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.
2020-06-20 08:39:49,787 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start
actor system at audit-dp03:0
2020-06-20 08:39:49,809 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2020-06-20 08:39:49,813 INFO akka.remote.Remoting
- Starting remoting
2020-06-20 08:39:49,824 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink-metrics@audit-dp03:58346]
2020-06-20 08:39:49,871 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system
started at akka.tcp://flink-metrics@audit-dp03:58346
2020-06-20 08:39:49,879 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/MetricQueryService_container_e07_1567067657620_0251_01_000005
.
2020-06-20 08:39:49,893 INFO org.apache.flink.runtime.blob.PermanentBlobCache
- Created BLOB cache storage directory
/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/blobStore-372d337e-6004-4681-8f2f-046e030713b9
2020-06-20 08:39:49,896 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Created BLOB cache storage directory
/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/blobStore-f9a39008-7a42-4a1b-810f-1a9e6fb7a8a5
2020-06-20 08:39:49,896 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting
TaskManager with ResourceID: container_e07_1567067657620_0251_01_000005
2020-06-20 08:39:50,035 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3452 GB (94.16% usable)
2020-06-20 08:39:50,036 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3448 GB (94.05% usable)
2020-06-20 08:39:50,036 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3448 GB (94.05% usable)
2020-06-20 08:39:50,036 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3449 GB (94.08% usable)
2020-06-20 08:39:50,036 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3449 GB (94.08% usable)
2020-06-20 08:39:50,037 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3448 GB (94.05% usable)
2020-06-20 08:39:50,037 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3451 GB (94.14% usable)
2020-06-20 08:39:50,037 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3450 GB (94.11% usable)
2020-06-20 08:39:50,037 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3446 GB (94.00% usable)
2020-06-20 08:39:50,038 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3449 GB (94.08% usable)
2020-06-20 08:39:50,038 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3450 GB (94.11% usable)
2020-06-20 08:39:50,038 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory
'/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251':
total 3666 GB, usable 3452 GB (94.16% usable)
2020-06-20 08:39:50,041 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-e468caf3-98df-467c-990a-8bef1d5eef0f
for spill files.
2020-06-20 08:39:50,041 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-d42e1f1b-cd8a-4cf1-bd14-817defea22cb
for spill files.
2020-06-20 08:39:50,041 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-103b1d5b-ff8e-44f0-8e67-ad76b78ad103
for spill files.
2020-06-20 08:39:50,041 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-8415eb91-532f-4641-a22b-ffd8376bcfb0
for spill files.
2020-06-20 08:39:50,042 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-9f1c03ee-aed2-4010-a4ae-f31a12ea0326
for spill files.
2020-06-20 08:39:50,042 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-6a2c3fea-d2db-40bf-b6ca-551a3fbccc05
for spill files.
2020-06-20 08:39:50,042 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-a34ee6fa-75c3-4795-b874-6b3afd830e15
for spill files.
2020-06-20 08:39:50,042 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-cd416cf8-c2d1-43aa-b27f-86dade9dcdb0
for spill files.
2020-06-20 08:39:50,042 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-85cb8ebc-79cc-4212-89b2-8510fae12fa2
for spill files.
2020-06-20 08:39:50,043 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-5536554e-5e92-4d0c-a7d7-6dee98afa596
for spill files.
2020-06-20 08:39:50,043 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-f980223d-9926-41f4-a7d0-dfd17c2856f4
for spill files.
2020-06-20 08:39:50,043 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-io-455b70e7-6a0f-4116-821d-71ffb7a3a4ff
for spill files.
2020-06-20 08:39:50,057 INFO
org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig
[server address: audit-dp03/10.191.31.70, server port: 0, ssl enabled: false,
memory segment size (bytes): 32768, transport type: NIO, number of server
threads: 1 (manual), number of client threads: 1 (manual), server connect
backlog: 0 (use Netty's default), client connect timeout (sec): 120,
send/receive buffer size (bytes): 0 (use Netty's default)]
2020-06-20 08:39:50,059 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-22b8da2b-c443-4f8f-b703-81f8e5d4efce
for spill files.
2020-06-20 08:39:50,059 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-603b75ba-eda5-497f-80dc-4f35c7a864b8
for spill files.
2020-06-20 08:39:50,060 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-245415dd-c240-491e-84e5-4e6a63d44ee6
for spill files.
2020-06-20 08:39:50,060 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-14930b27-6bd7-4757-934a-a3c1b09c7dad
for spill files.
2020-06-20 08:39:50,060 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-db177edb-85bc-406b-8513-10cd8905afbb
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-6a53c3e0-2f8f-4c7e-9cc1-5c9c541bf5a9
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-72467e77-53ab-4dfb-a8e3-080666fdffc9
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-e3e3b901-525b-4420-b35a-eb8df583b8a4
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-2c057bb5-3530-431e-bde9-d83f5845e0cb
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-582542d1-5752-4019-a022-dce7c48c5ed4
for spill files.
2020-06-20 08:39:50,061 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-24557d95-0dd1-4165-9230-b7e4daa24e40
for spill files.
2020-06-20 08:39:50,062 INFO
org.apache.flink.runtime.io.disk.FileChannelManagerImpl -
FileChannelManager uses directory
/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-netty-shuffle-1484555f-d3c7-4b02-bee9-381f58fd5d13
for spill files.
2020-06-20 08:39:50,508 INFO
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 588
MB for network buffer pool (number of memory segments: 18844, bytes per
segment: 32768).
2020-06-20 08:39:50,515 INFO
org.apache.flink.runtime.io.network.NettyShuffleEnvironment - Starting the
network environment and its components.
2020-06-20 08:39:50,553 INFO
org.apache.flink.runtime.io.network.netty.NettyClient - Successful
initialization (took 37 ms).
2020-06-20 08:39:50,597 INFO
org.apache.flink.runtime.io.network.netty.NettyServer - Successful
initialization (took 42 ms). Listening on SocketAddress /10.191.31.70:42207.
2020-06-20 08:39:50,599 INFO
org.apache.flink.runtime.taskexecutor.KvStateService - Starting the
kvState service and its components.
2020-06-20 08:39:50,599 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting
managed memory to 3870 MB, memory will be allocated lazily.
2020-06-20 08:39:50,610 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have
a max timeout of 10000 ms
2020-06-20 08:39:50,620 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.taskexecutor.TaskExecutor at
akka://flink/user/taskmanager_0 .
2020-06-20 08:39:50,633 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job
leader service.
2020-06-20 08:39:50,634 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data01/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-1fdc8487-93d5-4412-86cd-de10840d21c6
2020-06-20 08:39:50,634 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data02/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-80b703eb-e082-4170-a21c-ecf29d390f00
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data03/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-a8c233f2-0173-4872-b7ed-ba052e9d97ed
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data04/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-27a25ef0-9519-48af-950b-d761b30035df
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data05/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-e18525c3-fb6f-45f8-8ee4-70ddac044717
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data06/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-15e918b9-7183-4aa4-bf67-4b78d2364f73
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data07/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-816a2f73-15b1-43ad-ae0d-a385b3d25781
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data08/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-3ab75f48-70e6-4e39-b740-5b58702d8d3e
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data09/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-28f6b753-0b4a-4aba-b116-b38f552579eb
2020-06-20 08:39:50,635 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data10/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-97a6bc83-6d1f-40d8-9646-5df391550bd2
2020-06-20 08:39:50,636 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data11/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-8243e313-ed26-40be-aeef-a8a28200f487
2020-06-20 08:39:50,636 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/data12/hadoop/yarn/local/usercache/ocdp/appcache/application_1567067657620_0251/flink-dist-cache-a19547a4-d039-432e-8e83-843fa79460c2
2020-06-20 08:39:50,637 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to
ResourceManager
akka.tcp://flink@audit-dp04:57053/user/resourcemanager(00000000000000000000000000000000).
2020-06-20 08:39:50,798 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved
ResourceManager address, beginning registration
2020-06-20 08:39:50,798 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at
ResourceManager attempt 1 (timeout=100ms)
2020-06-20 08:39:50,835 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful
registration at resource manager
akka.tcp://flink@audit-dp04:57053/user/resourcemanager under registration id
167920416b0c54026975e6f9e569b432.
2020-06-20 08:39:50,854 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot
request 2731126d5f6739da10914421dd588ed6 for job
61c983cd478a19347676338a173dceb0 from resource manager with leader id
00000000000000000000000000000000.
2020-06-20 08:39:50,856 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot
for 2731126d5f6739da10914421dd588ed6.
2020-06-20 08:39:50,856 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job
61c983cd478a19347676338a173dceb0 for job leader monitoring.
2020-06-20 08:39:50,858 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register
at job manager akka.tcp://flink@audit-dp04:57053/user/jobmanager_0 with leader
id 00000000-0000-0000-0000-000000000000.
2020-06-20 08:39:50,875 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Resolved
JobManager address, beginning registration
2020-06-20 08:39:50,875 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Registration at
JobManager attempt 1 (timeout=100ms)
2020-06-20 08:39:50,891 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Successful
registration at job manager akka.tcp://flink@audit-dp04:57053/user/jobmanager_0
for job 61c983cd478a19347676338a173dceb0.
2020-06-20 08:39:50,892 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish
JobManager connection for job 61c983cd478a19347676338a173dceb0.
2020-06-20 08:39:50,895 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved
slots to the leader of job 61c983cd478a19347676338a173dceb0.
2020-06-20 08:39:50,905 INFO
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Activate slot
2731126d5f6739da10914421dd588ed6.
2020-06-20 08:39:52,127 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
map -> Filter (1/12).
2020-06-20 08:39:52,127 INFO org.apache.flink.runtime.taskmanager.Task
- map -> Filter (1/12) (1c5967676e1cde07b5d5b66440fcd577) switched
from CREATED to DEPLOYING.
2020-06-20 08:39:52,128 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task map -> Filter
(1/12) (1c5967676e1cde07b5d5b66440fcd577) [DEPLOYING]
2020-06-20 08:39:52,131 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task map -> Filter (1/12)
(1c5967676e1cde07b5d5b66440fcd577) [DEPLOYING].
2020-06-20 08:39:52,135 INFO org.apache.flink.runtime.blob.BlobClient
- Downloading
61c983cd478a19347676338a173dceb0/p-b06e800051b8703ec092a83b78a53b6737d303ea-20ef96f386cb633e49111720a1094657
from audit-dp04/10.191.31.71:39291
2020-06-20 08:39:52,143 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink (1/12).
2020-06-20 08:39:52,143 INFO org.apache.flink.runtime.taskmanager.Task
- OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink
(1/12) (3d379bfbd3807e5fc7b8833cfb22fe13) switched from CREATED to DEPLOYING.
2020-06-20 08:39:52,143 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task
OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink (1/12)
(3d379bfbd3807e5fc7b8833cfb22fe13) [DEPLOYING]
2020-06-20 08:39:52,144 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task OnlineOrderDispatchDurationDataProcess
-> Sink: ResultDataSink (1/12) (3d379bfbd3807e5fc7b8833cfb22fe13) [DEPLOYING].
2020-06-20 08:39:52,149 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
Source: source -> Filter -> Map (7/12).
2020-06-20 08:39:52,150 INFO org.apache.flink.runtime.taskmanager.Task
- Source: source -> Filter -> Map (7/12)
(3b6d7fff14d0bafbac085c4ae6c14580) switched from CREATED to DEPLOYING.
2020-06-20 08:39:52,150 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task Source:
source -> Filter -> Map (7/12) (3b6d7fff14d0bafbac085c4ae6c14580) [DEPLOYING]
2020-06-20 08:39:52,150 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task Source: source -> Filter -> Map (7/12)
(3b6d7fff14d0bafbac085c4ae6c14580) [DEPLOYING].
2020-06-20 08:39:53,064 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network:
OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink (1/12)
(3d379bfbd3807e5fc7b8833cfb22fe13) [DEPLOYING].
2020-06-20 08:39:53,064 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: Source: source -> Filter -> Map
(7/12) (3b6d7fff14d0bafbac085c4ae6c14580) [DEPLOYING].
2020-06-20 08:39:53,064 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: map -> Filter (1/12)
(1c5967676e1cde07b5d5b66440fcd577) [DEPLOYING].
2020-06-20 08:39:53,104 INFO org.apache.flink.runtime.taskmanager.Task
- Source: source -> Filter -> Map (7/12)
(3b6d7fff14d0bafbac085c4ae6c14580) switched from DEPLOYING to RUNNING.
2020-06-20 08:39:53,120 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Using
application-defined state backend: File State Backend (checkpoints:
'hdfs:/flink/checkpoints/OnlineOrderDispatchDuration', savepoints: 'null',
asynchronous: UNDEFINED, fileStateThreshold: -1)
2020-06-20 08:39:53,121 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Configuring
application-defined state backend with job/cluster config
2020-06-20 08:39:53,141 INFO org.apache.flink.runtime.taskmanager.Task
- OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink
(1/12) (3d379bfbd3807e5fc7b8833cfb22fe13) switched from DEPLOYING to RUNNING.
2020-06-20 08:39:53,142 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Using
application-defined state backend: File State Backend (checkpoints:
'hdfs:/flink/checkpoints/OnlineOrderDispatchDuration', savepoints: 'null',
asynchronous: UNDEFINED, fileStateThreshold: -1)
2020-06-20 08:39:53,142 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Configuring
application-defined state backend with job/cluster config
2020-06-20 08:39:53,155 INFO org.apache.flink.runtime.taskmanager.Task
- map -> Filter (1/12) (1c5967676e1cde07b5d5b66440fcd577) switched
from DEPLOYING to RUNNING.
2020-06-20 08:39:53,156 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Using
application-defined state backend: File State Backend (checkpoints:
'hdfs:/flink/checkpoints/OnlineOrderDispatchDuration', savepoints: 'null',
asynchronous: UNDEFINED, fileStateThreshold: -1)
2020-06-20 08:39:53,156 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Configuring
application-defined state backend with job/cluster config
2020-06-20 08:39:54,287 WARN org.apache.hadoop.io.retry.RetryInvocationHandler
- A failover has occurred since the start of
ClientNamenodeProtocolTranslatorPB.mkdirs over audit-dp02/10.191.31.69:8020
2020-06-20 08:39:54,287 WARN org.apache.hadoop.io.retry.RetryInvocationHandler
- A failover has occurred since the start of
ClientNamenodeProtocolTranslatorPB.mkdirs over audit-dp02/10.191.31.69:8020
2020-06-20 08:39:54,420 INFO
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction -
FlinkKafkaProducer 0/12 - restoring state
2020-06-20 08:39:54,421 INFO
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction -
FlinkKafkaProducer 0/12 committed recovered transaction
TransactionHolder{handle=KafkaTransactionState [transactionalId=null,
producerId=-1, epoch=-1], transactionStartTime=1592440442910}
2020-06-20 08:39:54,421 INFO
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction -
FlinkKafkaProducer 0/12 aborted recovered transaction
TransactionHolder{handle=KafkaTransactionState [transactionalId=null,
producerId=-1, epoch=-1], transactionStartTime=1592441042958}
2020-06-20 08:39:54,421 INFO
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction -
FlinkKafkaProducer 0/12 - no state to restore
2020-06-20 08:39:54,445 INFO org.apache.kafka.clients.producer.ProducerConfig
- ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [dn49.hadoop.unicom:6667, dn50.hadoop.unicom:6667,
dn51.hadoop.unicom:6667, dn54.hadoop.unicom:6667, dn55.hadoop.unicom:6667,
dn56.hadoop.unicom:6667]
buffer.memory = 33554432
client.dns.lookup = default
client.id =
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class
org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 3600000
transactional.id = null
value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
2020-06-20 08:39:54,504 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka version: 2.2.0
2020-06-20 08:39:54,504 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka commitId: 05fcfde8f69b0349
2020-06-20 08:39:54,506 INFO
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer - Starting
FlinkKafkaInternalProducer (1/12) to produce into default topic
UA_RESULT_DATA_OF_FLINK
2020-06-20 08:39:54,605 INFO
org.apache.flink.runtime.state.heap.HeapKeyedStateBackend - Initializing
heap keyed state backend with stream factory.
2020-06-20 08:39:54,755 INFO org.apache.kafka.clients.Metadata
- Cluster ID: oaeNn86gQ-Wd3dXwYN8aFA
2020-06-20 08:39:54,883 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- class
org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition does
not contain a setter for field topic
2020-06-20 08:39:54,883 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- Class class
org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition
cannot be used as a POJO type because not all fields are valid POJO fields, and
must be processed as GenericType. Please read the Flink documentation on "Data
Types & Serialization" for details of the effect on performance.
2020-06-20 08:39:54,889 INFO
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Consumer
subtask 6 restored state: {KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=0}=25232342, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=1}=25584491, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=2}=25473334, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=3}=26069261, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=4}=25987404, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=5}=25873696, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=6}=25762624, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=7}=26121096, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=8}=25410865, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=9}=26016097, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=10}=25957727, KafkaTopicPartition{topic='UA_DATA_FOR_SPARK',
partition=11}=25770606}.
2020-06-20 08:39:54,898 INFO org.apache.kafka.clients.consumer.ConsumerConfig
- ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [dn49.hadoop.unicom:6667, dn50.hadoop.unicom:6667,
dn51.hadoop.unicom:6667, dn54.hadoop.unicom:6667, dn55.hadoop.unicom:6667,
dn56.hadoop.unicom:6667]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = OnlineOrderDispatchDuration_20200401
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-06-20 08:39:54,927 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka version: 2.2.0
2020-06-20 08:39:54,928 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka commitId: 05fcfde8f69b0349
2020-06-20 08:39:54,940 INFO
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Consumer
subtask 6 will start reading 1 partitions with offsets in restored state:
{KafkaTopicPartition{topic='UA_DATA_FOR_SPARK', partition=7}=26121096}
2020-06-20 08:39:54,943 INFO
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Consumer
subtask 6 creating fetcher with offsets
{KafkaTopicPartition{topic='UA_DATA_FOR_SPARK', partition=7}=26121096}.
2020-06-20 08:39:54,953 INFO org.apache.kafka.clients.consumer.ConsumerConfig
- ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [dn49.hadoop.unicom:6667, dn50.hadoop.unicom:6667,
dn51.hadoop.unicom:6667, dn54.hadoop.unicom:6667, dn55.hadoop.unicom:6667,
dn56.hadoop.unicom:6667]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = OnlineOrderDispatchDuration_20200401
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-06-20 08:39:54,959 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka version: 2.2.0
2020-06-20 08:39:54,959 INFO org.apache.kafka.common.utils.AppInfoParser
- Kafka commitId: 05fcfde8f69b0349
2020-06-20 08:39:54,962 INFO org.apache.kafka.clients.consumer.KafkaConsumer
- [Consumer clientId=consumer-2,
groupId=OnlineOrderDispatchDuration_20200401] Subscribed to partition(s):
UA_DATA_FOR_SPARK-7
2020-06-20 08:39:54,974 INFO org.apache.kafka.clients.Metadata
- Cluster ID: oaeNn86gQ-Wd3dXwYN8aFA
2020-06-20 08:39:55,385 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to cancel task map -> Filter (1/12)
(1c5967676e1cde07b5d5b66440fcd577).
2020-06-20 08:39:55,385 INFO org.apache.flink.runtime.taskmanager.Task
- map -> Filter (1/12) (1c5967676e1cde07b5d5b66440fcd577) switched
from RUNNING to CANCELING.
2020-06-20 08:39:55,386 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code map -> Filter (1/12)
(1c5967676e1cde07b5d5b66440fcd577).
2020-06-20 08:39:55,391 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to cancel task OnlineOrderDispatchDurationDataProcess
-> Sink: ResultDataSink (1/12) (3d379bfbd3807e5fc7b8833cfb22fe13).
2020-06-20 08:39:55,391 INFO org.apache.flink.runtime.taskmanager.Task
- OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink
(1/12) (3d379bfbd3807e5fc7b8833cfb22fe13) switched from RUNNING to CANCELING.
2020-06-20 08:39:55,391 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code
OnlineOrderDispatchDurationDataProcess -> Sink: ResultDataSink (1/12)
(3d379bfbd3807e5fc7b8833cfb22fe13).
2020-06-20 08:39:55,395 INFO org.apache.flink.runtime.taskmanager.Task
- Attempting to cancel task Source: source -> Filter -> Map (7/12)
(3b6d7fff14d0bafbac085c4ae6c14580).
2020-06-20 08:39:55,395 INFO org.apache.flink.runtime.taskmanager.Task
- Source: source -> Filter -> Map (7/12)
(3b6d7fff14d0bafbac085c4ae6c14580) switched from RUNNING to CANCELING.
2020-06-20 08:39:55,395 INFO org.apache.flink.runtime.taskmanager.Task
- Triggering cancellation of task code Source: source -> Filter ->
Map (7/12) (3b6d7fff14d0bafbac085c4ae6c14580).
2020-06-20 08:39:55,399 WARN
org.apache.flink.streaming.api.operators.BackendRestorerProcedure - Exception
while restoring keyed state backend for
StreamMap_23ab3a59b17e9c45f95cff4c728611fc_(1/12) from alternative (1/1), will
retry while more alternatives are available.
org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to
restore heap backend
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:116)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createKeyedStateBackend(FsStateBackend.java:529)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:253)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Stream closed
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:876)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:945)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:741)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
org.apache.flink.runtime.fs.hdfs.HadoopDataInputStream.read(HadoopDataInputStream.java:84)
at
org.apache.flink.core.fs.FSDataInputStreamWrapper.read(FSDataInputStreamWrapper.java:51)
at
org.apache.flink.runtime.util.ForwardingInputStream.read(ForwardingInputStream.java:41)
at java.io.DataInputStream.readBoolean(DataInputStream.java:242)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:405)
at
org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:151)
at
org.apache.flink.api.common.typeutils.base.MapSerializer.deserialize(MapSerializer.java:151)
at
org.apache.flink.api.common.typeutils.base.MapSerializer.deserialize(MapSerializer.java:43)
at
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders.lambda$createV2PlusReader$0(StateTableByKeyGroupReaders.java:77)
at
org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readKeyGroupStateData(HeapRestoreOperation.java:293)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:254)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:153)
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:114)
... 12 more
