Hi Jorn, There are no more logs . Attaching yarn aggregated logs for first problem . For second one job is not even getting submitted.
- Sohi On Sun, Dec 9, 2018 at 2:13 PM Jörn Franke <jornfra...@gmail.com> wrote: > Can you check the Flink log files? You should get there a better > description of the error. > > > Am 08.12.2018 um 18:15 schrieb sohimankotia <sohimanko...@gmail.com>: > > > > Hi , > > > > I have installed flink-1.7.0 Hadoop 2.7 scala 2.11 . We are using > > hortonworks hadoop distribution.(hdp/2.6.1.0-129/) > > > > *Flink lib folder looks like :* > > > > > > -rw-r--r-- 1 hdfs hadoop 93184216 Nov 29 02:15 flink-dist_2.11-1.7.0.jar > > -rw-r--r-- 1 hdfs hadoop 79219 Nov 29 03:33 > > flink-hadoop-compatibility_2.11-1.7.0.jar > > -rw-r--r-- 1 hdfs hadoop 141881 Nov 29 02:13 > flink-python_2.11-1.7.0.jar > > -rw-r--r-- 1 hdfs hadoop 489884 Nov 28 23:01 log4j-1.2.17.jar > > -rw-r--r-- 1 hdfs hadoop 9931 Nov 28 23:01 slf4j-log4j12-1.7.15.j > > > > *My code :* > > > > ExecutionEnvironment env = > > ExecutionEnvironment.getExecutionEnvironment(); > > > > String p = args[0]; > > > > > > Job job = Job.getInstance(); > > SequenceFileInputFormat<Text, BytesWritable> inputFormat = new > > SequenceFileInputFormat<>(); > > > > job.getConfiguration().setBoolean(FileInputFormat.INPUT_DIR_RECURSIVE, > > true); > > final HadoopInputFormat<Text, BytesWritable> hInputEvents = > > HadoopInputs.readHadoopFile(inputFormat, Text.class, > BytesWritable.class, p, > > job); > > org.apache.flink.configuration.Configuration fileReadConfig = new > > org.apache.flink.configuration.Configuration(); > > > > env.createInput(hInputEvents) > > .output(new PrintingOutputFormat<>()); > > > > > > *pom.xml* > > > > flink.version = 1.7.0 > > > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-java</artifactId> > > <version>${flink.version}</version> > > <scope>provided</scope> > > </dependency> > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-clients_2.11</artifactId> > > <version>${flink.version}</version> > > <scope>provided</scope> > > </dependency> > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-streaming-java_2.11</artifactId> > > <version>${flink.version}</version> > > <scope>provided</scope> > > </dependency> > > > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-hadoop-compatibility_2.11</artifactId> > > <version>${flink.version}</version> > > <scope>provided</scope> > > </dependency> > > > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-shaded-hadoop2</artifactId> > > <version>${flink.version}</version> > > <scope>provided</scope> > > </dependency> > > > > * > > in script :* > > > > > > > > export HADOOP_CONF_DIR=/etc/hadoop/conf > > export HADOOP_CLASSPATH="/usr/hdp/2.6.1.0-129/hadoop/hadoop-*":`hadoop > > classpath` > > > > echo ${HADOOP_CLASSPATH} > > > > PARALLELISM=1 > > JAR_PATH="jar" > > CLASS_NAME="CLASS_NAME" > > NODES=1 > > SLOTS=1 > > MEMORY_PER_NODE=2048 > > QUEUE="default" > > NAME="sample" > > > > IN="input-file-path" > > > > > > /home/hdfs/flink-1.7.0/bin/flink run -m yarn-cluster -yn ${NODES} -yqu > > ${QUEUE} -ys ${SLOTS} -ytm ${MEMORY_PER_NODE} --parallelism > ${PARALLELISM} > > -ynm ${NAME} -c ${CLASS_NAME} ${JAR_PATH} ${IN} > > > > > > *where classpath is printing:* > > > > > /usr/hdp/2.6.1.0-129/hadoop/hadoop-*:/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*:/usr/hdp/ > 2.6.1. > 0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf > > > > > > But I am getting class not found error for hadoop related jar . Error is > > attached . > > > > > > error.txt > > < > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t894/error.txt> > > > *Another Problem :* > > > > If i added hadoop shaded jar in lib folder > > > > > > -rw-r--r-- 1 hdfs hadoop 93184216 Nov 29 02:15 flink-dist_2.11-1.7.0.jar > > -rw-r--r-- 1 hdfs hadoop 79219 Nov 29 03:33 > > flink-hadoop-compatibility_2.11-1.7.0.jar > > -rw-r--r-- 1 hdfs hadoop 141881 Nov 29 02:13 > flink-python_2.11-1.7.0.jar > > *-rw-r--r-- 1 hdfs hadoop 41130742 Dec 8 22:38 > > flink-shaded-hadoop2-uber-1.7.0.jar* > > -rw-r--r-- 1 hdfs hadoop 489884 Nov 28 23:01 log4j-1.2.17.jar > > -rw-r--r-- 1 hdfs hadoop 9931 Nov 28 23:01 slf4j-log4j12-1.7.15.jar > > > > I am getting following error. And this is happening for all version > greater > > than 1.4.2 . > > > > java.lang.IllegalAccessError: tried to access method > > > org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; > > from class > > org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider > > at > > > org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75) > > at > > > org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163) > > at > org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) > > at > > > org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) > > at > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) > > at > > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > > at > > > org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:985) > > at > > > org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:273) > > at > > > org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:451) > > at > > > org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:96) > > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:224) > > > > > > Thanks in advance . > > > > > > > > > > > > -- > > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
Container: container_e186_1541304140537_100017_01_000001 on nn-hw-m1d23.dbp.name.in_45454_1544345374804 LogAggregationType: AGGREGATED ============================================================================================================ LogType:directory.info Log Upload Time:Sun Dec 09 14:19:34 +0530 2018 LogLength:2223 Log Contents: ls -l: total 40 -rw------- 1 yarn hadoop 71 Dec 9 14:19 container_tokens lrwxrwxrwx 1 yarn hadoop 168 Dec 9 14:19 flink-conf.yaml -> /grid/12/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/11/application_1541304140537_100017-flink-conf.yaml1533676801661714355.tmp lrwxrwxrwx 1 yarn hadoop 121 Dec 9 14:19 flink.jar -> /grid/6/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/16/flink-dist_2.11-1.7.0.jar -rwx------ 1 yarn hadoop 10659 Dec 9 14:19 launch_container.sh drwxr-x--- 2 yarn hadoop 4096 Dec 9 14:19 lib lrwxrwxrwx 1 yarn hadoop 112 Dec 9 14:19 log4j.properties -> /grid/5/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/15/log4j.properties lrwxrwxrwx 1 yarn hadoop 107 Dec 9 14:19 logback.xml -> /grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/12/logback.xml drwxr-x--- 2 yarn hadoop 4096 Dec 9 14:19 tmp find -L . -maxdepth 5 -ls: 5111823 4 drwxr-x--- 4 yarn hadoop 4096 Dec 9 14:19 . 5111826 4 -rw------- 1 yarn hadoop 71 Dec 9 14:19 ./container_tokens 139198601 4 -r-x------ 1 yarn hadoop 292 Dec 9 14:19 ./flink-conf.yaml 168427613 4 -r-x------ 1 yarn hadoop 2331 Dec 9 14:19 ./logback.xml 5111824 4 drwxr-x--- 2 yarn hadoop 4096 Dec 9 14:19 ./tmp 5111825 12 -rwx------ 1 yarn hadoop 10659 Dec 9 14:19 ./launch_container.sh 133300401 4 -r-x------ 1 yarn hadoop 1939 Dec 9 14:19 ./log4j.properties 54001673 91004 -r-x------ 1 yarn hadoop 93184216 Dec 9 14:19 ./flink.jar 5111828 4 drwxr-x--- 2 yarn hadoop 4096 Dec 9 14:19 ./lib 241041669 140 -r-x------ 1 yarn hadoop 141881 Dec 9 14:19 ./lib/flink-python_2.11-1.7.0.jar 50725055 80 -r-x------ 1 yarn hadoop 79219 Dec 9 14:19 ./lib/flink-hadoop-compatibility_2.11-1.7.0.jar 11010215 12 -r-x------ 1 yarn hadoop 9931 Dec 9 14:19 ./lib/slf4j-log4j12-1.7.15.jar 120586377 480 -r-x------ 1 yarn hadoop 489884 Dec 9 14:19 ./lib/log4j-1.2.17.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Sun Dec 09 14:19:34 +0530 2018 LogLength:10659 Log Contents: #!/bin/bash export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/mnt/vol1/jdk1.8.0_144/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export MAX_APP_ATTEMPTS="1" export _SLOTS="1" export JAVA_HOME=${JAVA_HOME:-"/mnt/vol1/jdk1.8.0_144/"} export _CLIENT_HOME_DIR="hdfs://hwcluster01/user/hdfs" export LANG="en_US.UTF-8" export APP_SUBMIT_TIME_ENV="1544345364329" export NM_HOST="nn-hw-m1d23.dbp.name.in" export _APP_ID="application_1541304140537_100017" export HADOOP_USER_NAME="hdfs" export LOGNAME="hdfs" export JVM_PID="$$" export _DETACHED="false" export PWD="/grid/8/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001" export LOCAL_DIRS="/grid/1/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/10/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/11/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/12/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/3/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/4/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/5/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/6/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/7/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/8/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/9/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017" export APPLICATION_WEB_PROXY_BASE="/proxy/application_1541304140537_100017" export _DYNAMIC_PROPERTIES="" export NM_HTTP_PORT="8042" export LOG_DIRS="/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/10/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/11/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/12/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/2/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/4/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/5/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/6/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/7/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/8/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001,/grid/9/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export _CLIENT_SHIP_FILES="logback.xml=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/logback.xml,log4j.properties=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/log4j.properties,lib/flink-python_2.11-1.7.0.jar=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/lib/flink-python_2.11-1.7.0.jar,lib/log4j-1.2.17.jar=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/lib/log4j-1.2.17.jar,lib/flink-hadoop-compatibility_2.11-1.7.0.jar=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/lib/flink-hadoop-compatibility_2.11-1.7.0.jar,lib/slf4j-log4j12-1.7.15.jar=hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/lib/slf4j-log4j12-1.7.15.jar," export NM_PORT="45454" export USER="hdfs" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="lib/flink-hadoop-compatibility_2.11-1.7.0.jar:lib/flink-python_2.11-1.7.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*" export _FLINK_YARN_FILES="hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017" export _ZOOKEEPER_NAMESPACE="application_1541304140537_100017" export HADOOP_TOKEN_FILE_LOCATION="/grid/8/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export _FLINK_JAR_PATH="hdfs://hwcluster01/user/hdfs/.flink/application_1541304140537_100017/flink-dist_2.11-1.7.0.jar" export LOCAL_USER_DIRS="/grid/1/hadoop/yarn/local/usercache/hdfs/,/grid/10/hadoop/yarn/local/usercache/hdfs/,/grid/11/hadoop/yarn/local/usercache/hdfs/,/grid/12/hadoop/yarn/local/usercache/hdfs/,/grid/2/hadoop/yarn/local/usercache/hdfs/,/grid/3/hadoop/yarn/local/usercache/hdfs/,/grid/4/hadoop/yarn/local/usercache/hdfs/,/grid/5/hadoop/yarn/local/usercache/hdfs/,/grid/6/hadoop/yarn/local/usercache/hdfs/,/grid/7/hadoop/yarn/local/usercache/hdfs/,/grid/8/hadoop/yarn/local/usercache/hdfs/,/grid/9/hadoop/yarn/local/usercache/hdfs/" export HADOOP_HOME="/usr/hdp/2.6.1.0-129/hadoop" export _FLINK_CLASSPATH="lib/flink-hadoop-compatibility_2.11-1.7.0.jar:lib/flink-python_2.11-1.7.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml:" export _CLIENT_TM_COUNT="1" export _CLIENT_TM_MEMORY="2048" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_e186_1541304140537_100017_01_000001" export MALLOC_ARENA_MAX="4" ln -sf "/grid/12/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/11/application_1541304140537_100017-flink-conf.yaml1533676801661714355.tmp" "flink-conf.yaml" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi mkdir -p lib hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/7/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/17/log4j-1.2.17.jar" "lib/log4j-1.2.17.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi mkdir -p lib hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/4/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/14/flink-python_2.11-1.7.0.jar" "lib/flink-python_2.11-1.7.0.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/6/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/16/flink-dist_2.11-1.7.0.jar" "flink.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi mkdir -p lib hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/3/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/13/flink-hadoop-compatibility_2.11-1.7.0.jar" "lib/flink-hadoop-compatibility_2.11-1.7.0.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/12/logback.xml" "logback.xml" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi mkdir -p lib hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/11/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/10/slf4j-log4j12-1.7.15.jar" "lib/slf4j-log4j12-1.7.15.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/grid/5/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/15/log4j.properties" "log4j.properties" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/launch_container.sh" chmod 640 "/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" ls -l 1>>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" find -L . -maxdepth 5 -ls 1>>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/grid/1/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/directory.info" exec /bin/bash -c "$JAVA_HOME/bin/java -Xmx424m -Dlog.file="/grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint 1> /grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.out 2> /grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.err" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:jobmanager.err Log Upload Time:Sun Dec 09 14:19:34 +0530 2018 LogLength:533 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/grid/11/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.1.0-129/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] End of LogType:jobmanager.err LogType:jobmanager.log Log Upload Time:Sun Dec 09 14:19:34 +0530 2018 LogLength:45929 Log Contents: 2018-12-09 14:19:28,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-12-09 14:19:28,622 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.7.0, Rev:49da9f9, Date:28.11.2018 @ 17:59:06 UTC) 2018-12-09 14:19:28,623 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn 2018-12-09 14:19:29,031 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-12-09 14:19:29,067 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hdfs 2018-12-09 14:19:29,067 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.144-b01 2018-12-09 14:19:29,067 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes 2018-12-09 14:19:29,067 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /mnt/vol1/jdk1.8.0_144/ 2018-12-09 14:19:29,068 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.7.3.2.6.1.0-129 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.log 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none) 2018-12-09 14:19:29,069 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-hadoop-compatibility_2.11-1.7.0.jar:lib/flink-python_2.11-1.7.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.1.0-129-tests.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/joda-time-2.9.4.jar:/usr/hdp/current/hadoop-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.1.0-129-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.1.0-129-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.1.0-129.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar 2018-12-09 14:19:29,070 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-12-09 14:19:29,070 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2018-12-09 14:19:29,135 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hdfs Yarn client user obtainer: hdfs 2018-12-09 14:19:29,140 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-12-09 14:19:29,140 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1541304140537_100017 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 2048m 2018-12-09 14:19:29,141 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m 2018-12-09 14:19:29,160 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /grid/1/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/10/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/11/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/12/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/3/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/4/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/5/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/6/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/7/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/8/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017,/grid/9/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017 2018-12-09 14:19:29,187 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint. 2018-12-09 14:19:29,187 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2018-12-09 14:19:29,261 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hdfs (auth:SIMPLE) 2018-12-09 14:19:29,276 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2018-12-09 14:19:29,429 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at nn-hw-m1d23.dbp.name.in:0 2018-12-09 14:19:29,804 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2018-12-09 14:19:29,895 INFO akka.remote.Remoting - Starting remoting 2018-12-09 14:19:30,072 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121] 2018-12-09 14:19:30,080 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121 2018-12-09 14:19:30,100 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /grid/12/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/blobStore-5308aebb-1216-439e-ba5e-aafc4822af44 2018-12-09 14:19:30,111 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:22767 - max concurrent requests: 50 - max backlog: 1000 2018-12-09 14:19:30,157 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2018-12-09 14:19:30,158 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at nn-hw-m1d23.dbp.name.in:0 2018-12-09 14:19:30,176 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2018-12-09 14:19:30,185 INFO akka.remote.Remoting - Starting remoting 2018-12-09 14:19:30,195 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metr...@nn-hw-m1d23.dbp.name.in:15452] 2018-12-09 14:19:30,196 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metr...@nn-hw-m1d23.dbp.name.in:15452 2018-12-09 14:19:30,202 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /grid/1/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/executionGraphStore-12e37892-5523-4e94-94c2-005875fcce45, expiration time 3600000, maximum cache size 52428800 bytes. 2018-12-09 14:19:30,225 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /grid/11/hadoop/yarn/local/usercache/hdfs/appcache/application_1541304140537_100017/blobStore-bf84b253-7cac-41a4-9adf-8388f571ab9b 2018-12-09 14:19:30,248 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-6345fac2-a787-46a3-be09-4ab1cf646849/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2018-12-09 14:19:30,260 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-6345fac2-a787-46a3-be09-4ab1cf646849/flink-web-upload for file uploads. 2018-12-09 14:19:30,264 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. 2018-12-09 14:19:30,510 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.log 2018-12-09 14:19:30,510 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /grid/3/hadoop/yarn/log/application_1541304140537_100017/container_e186_1541304140537_100017_01_000001/jobmanager.out 2018-12-09 14:19:30,679 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at nn-hw-m1d23.dbp.name.in:26980 2018-12-09 14:19:30,679 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://nn-hw-m1d23.dbp.name.in:26980 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2018-12-09 14:19:30,679 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://nn-hw-m1d23.dbp.name.in:26980. 2018-12-09 14:19:30,745 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2018-12-09 14:19:30,797 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . 2018-12-09 14:19:31,022 INFO org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider - Looking for the active RM in [rm1, rm2]... 2018-12-09 14:19:31,289 INFO org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider - Found active RM [rm2] 2018-12-09 14:19:31,291 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2018-12-09 14:19:31,303 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2018-12-09 14:19:31,305 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000 2018-12-09 14:19:31,306 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2018-12-09 14:19:31,335 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000 2018-12-09 14:19:31,341 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs. 2018-12-09 14:19:32,467 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 53e77d649a08e1274a2bcd15ce979515 (Hadoop WordCount). 2018-12-09 14:19:32,525 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2018-12-09 14:19:32,533 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job Hadoop WordCount (53e77d649a08e1274a2bcd15ce979515). 2018-12-09 14:19:32,545 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy NoRestartStrategy for Hadoop WordCount (53e77d649a08e1274a2bcd15ce979515). 2018-12-09 14:19:32,549 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/6a0b9952-5c9f-43e0-91c6-c260a9d73dbe . 2018-12-09 14:19:32,572 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2018-12-09 14:19:32,598 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job Hadoop WordCount (53e77d649a08e1274a2bcd15ce979515). 2018-12-09 14:19:32,609 ERROR org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Failed to submit job 53e77d649a08e1274a2bcd15ce979515. java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:176) at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:1058) at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:308) at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34) ... 7 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSource (at createInput(ExecutionEnvironment.java:548) (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat))': Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@4441d567) failed: org/apache/hadoop/mapreduce/JobContext at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1166) at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1146) at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296) at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157) ... 10 more Caused by: java.lang.Exception: Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@4441d567) failed: org/apache/hadoop/mapreduce/JobContext at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:66) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) ... 15 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/JobContext at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2583) at java.lang.Class.getDeclaredField(Class.java:2068) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498) at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459) at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288) at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:63) ... 16 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.JobContext at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 44 more 2018-12-09 14:19:32,612 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation error: Unhandled exception. org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job. at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$submitJob$2(Dispatcher.java:267) at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) at akka.actor.Actor$class.aroundReceive(Actor.scala:502) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) ... 4 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:176) at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:1058) at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:308) at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34) ... 7 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSource (at createInput(ExecutionEnvironment.java:548) (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat))': Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@4441d567) failed: org/apache/hadoop/mapreduce/JobContext at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1166) at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1146) at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296) at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157) ... 10 more Caused by: java.lang.Exception: Deserializing the InputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat@4441d567) failed: org/apache/hadoop/mapreduce/JobContext at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:66) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) ... 15 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/JobContext at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2583) at java.lang.Class.getDeclaredField(Class.java:2068) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498) at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459) at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288) at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:63) ... 16 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.JobContext at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 44 more 2018-12-09 14:19:32,713 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping dispatcher akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121/user/dispatcher. 2018-12-09 14:19:32,713 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopping all currently running jobs of dispatcher akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121/user/dispatcher. 2018-12-09 14:19:32,719 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator - Shutting down stack trace sample coordinator. 2018-12-09 14:19:32,722 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Stopped dispatcher akka.tcp://fl...@nn-hw-m1d23.dbp.name.in:5121/user/dispatcher. 2018-12-09 14:19:32,723 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Shutting YarnSessionClusterEntrypoint down with application status SUCCEEDED. Diagnostics null. 2018-12-09 14:19:32,723 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shutting down rest endpoint. 2018-12-09 14:19:32,774 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Removing cache directory /tmp/flink-web-6345fac2-a787-46a3-be09-4ab1cf646849/flink-web-ui 2018-12-09 14:19:32,774 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://nn-hw-m1d23.dbp.name.in:26980 lost leadership 2018-12-09 14:19:32,775 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Shut down complete. 2018-12-09 14:19:32,776 INFO org.apache.flink.yarn.YarnResourceManager - Shut down cluster because application is in SUCCEEDED, diagnostics null. 2018-12-09 14:19:32,778 INFO org.apache.flink.yarn.YarnResourceManager - Unregister application from the YARN Resource Manager with final status SUCCEEDED. 2018-12-09 14:19:32,809 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for application to be successfully unregistered. 2018-12-09 14:19:33,471 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2018-12-09 14:19:33,511 INFO org.apache.hadoop.yarn.client.api.impl.NMClientImpl - Clean up running containers on stop. 2018-12-09 14:19:33,511 INFO org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:276) 2018-12-09 14:19:33,511 INFO org.apache.hadoop.yarn.client.api.impl.NMClientImpl - Running containers cleaned up. Stopping NM proxies. 2018-12-09 14:19:33,512 INFO org.apache.hadoop.yarn.client.api.impl.NMClientImpl - Stopped all proxies. 2018-12-09 14:19:33,512 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Closing the SlotManager. 2018-12-09 14:19:33,512 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Suspending the SlotManager. 2018-12-09 14:19:33,514 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:22767 2018-12-09 14:19:33,514 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache 2018-12-09 14:19:33,519 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. 2018-12-09 14:19:33,524 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Stopping SlotPool. 2018-12-09 14:19:33,525 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job Hadoop WordCount(53e77d649a08e1274a2bcd15ce979515). 2018-12-09 14:19:33,525 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 2018-12-09 14:19:33,527 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 2018-12-09 14:19:33,530 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 2018-12-09 14:19:33,533 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 2018-12-09 14:19:33,555 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 2018-12-09 14:19:33,555 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 2018-12-09 14:19:33,576 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. 2018-12-09 14:19:33,577 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Terminating cluster entrypoint process YarnSessionClusterEntrypoint with exit code 0. End of LogType:jobmanager.log LogType:jobmanager.out Log Upload Time:Sun Dec 09 14:19:34 +0530 2018 LogLength:0 Log Contents: End of LogType:jobmanager.out