????????????,????????bin/flink run -m yarn-cluster -yjm 660 -ytm 1024 -ys 1
-yqu xjia_queue -ynm Test_Demo02 -c com.dwd.test_main.Test_Demo02
-Drest.port="8067" /opt/module/flink1.12/xjia_lib/xjia_shuyun-6.0.jar
WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil
[] - The configuration directory
('/home/xjia/opt/module/flink1.12/conf') already contains a LOG4J config
file.If you want to use logback, then please delete or rename the log
configuration file.
2021-01-02 14:55:11,888 INFO org.apache.hadoop.yarn.client.RMProxy
[]
- Connecting to ResourceManager at /0.0.0.0:8032
2021-01-02 14:55:12,056 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the
location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-01-02 14:55:12,206 WARN
org.apache.flink.yarn.YarnClusterDescriptor
[] - Neither the HADOOP_CONF_DIR nor the
YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of
these to be set to properly load the Hadoop configuration for accessing YARN.
2021-01-02 14:55:12,237 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured JobManager memory is 660 MB.
YARN will allocate 1024 MB to make up an integer multiple of its minimum
allocation memory (1024 MB, configured via
'yarn.scheduler.minimum-allocation-mb'). The extra 364 MB may not be used by
Flink.
2021-01-02 14:55:12,238 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Cluster specification:
ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024,
slotsPerTaskManager=1}
2021-01-02 14:55:12,458 WARN
org.apache.flink.yarn.YarnClusterDescriptor
[] - The file system scheme is 'file'. This
indicates that the specified Hadoop configuration path is wrong and the system
is using the default Hadoop configuration values.The Flink YARN client needs to
store its files in a distributed file system
2021-01-02 14:55:13,775 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Submitting application master
application_1609403978979_0043
2021-01-02 14:55:14,025 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
[] - Submitted application application_1609403978979_0043
2021-01-02 14:55:14,026 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Waiting for the cluster to be allocated
2021-01-02 14:55:14,029 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Deploying cluster, current state ACCEPTED
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Failed to execute sql
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
Caused by: org.apache.flink.table.api.TableException: Failed to execute sql
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:696)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:759)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:665)
at com.dwd.test_main.Test_Demo02$.main(Test_Demo02.scala:50)
at com.dwd.test_main.Test_Demo02.main(Test_Demo02.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:316)
... 11 more
Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could
not deploy Yarn job cluster.
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:460)
at
org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1940)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128)
at
org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:57)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:680)
... 20 more
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException:
The YARN application unexpectedly switched to state FAILED during
deployment.
Diagnostics from YARN: Application application_1609403978979_0043 failed 2
times in previous 10000 milliseconds due to AM Container for
appattempt_1609403978979_0043_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2021-01-02 14:55:14.203]File
file:/home/xjia/.flink/application_1609403978979_0043/lib/flink-sql-parquet_2.11-1.12.0.jar
does not exist
java.io.FileNotFoundException: File
file:/home/xjia/.flink/application_1609403978979_0043/lib/flink-sql-parquet_2.11-1.12.0.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at
org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:236)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:224)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
For more detailed output, check the application tracking page:
http://hadoop31:8088/cluster/app/application_1609403978979_0043 Then click on
links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further
investigate the issue:
yarn logs -applicationId application_1609403978979_0043
at
org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1078)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:453)
... 25 more
2021-01-02 14:55:14,405 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Cancelling deployment from Deployment Failure
Hook
2021-01-02 14:55:14,413 INFO org.apache.hadoop.yarn.client.RMProxy
[]
- Connecting to ResourceManager at /0.0.0.0:8032
2021-01-02 14:55:14,418 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Killing YARN application
2021-01-02 14:55:14,432 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
[] - Killed application application_1609403978979_0043
2021-01-02 14:55:14,533 INFO
org.apache.flink.yarn.YarnClusterDescriptor
[] - Deleting files in
file:/home/xjia/.flink/application_1609403978979_0043.
