Re: Flink Memory analyze on AWS EMR
gt;> >>>>>>> Thanks a lot for posting the full output. >>>>>>> >>>>>>> It seems that Flink is passing an invalid list of arguments to the >>>>>>> JVM. >>>>>>> Can you >>>>>>> - set the root log level in conf/log4j-yarn-session.properties to >>>>>>> DEBUG >>>>>>> - then launch the YARN session >>>>>>> - share the log file of the yarn session on the mailing list? >>>>>>> >>>>>>> I'm particularly interested in the line printed here, as it shows >>>>>>> the JVM invocation. >>>>>>> >>>>>>> https://github.com/apache/flink/blob/release-1.6/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L1630 >>>>>>> >>>>>>> >>>>>>> On Mon, May 11, 2020 at 9:56 PM Jacky D >>>>>>> wrote: >>>>>>> >>>>>>>> Hi，Robert >>>>>>>> >>>>>>>> Yes , I tried to retrieve more log info from yarn UI , the full >>>>>>>> logs showing below , this happens when I try to create a flink yarn >>>>>>>> session >>>>>>>> on emr when set up jitwatch configuration . >>>>>>>> >>>>>>>> 2020-05-11 19:06:09,552 ERROR >>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error >>>>>>>> while >>>>>>>> running the Flink Yarn session. >>>>>>>> java.lang.reflect.UndeclaredThrowableException >>>>>>>> at >>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) >>>>>>>> at >>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>>>>>>> at >>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813) >>>>>>>> Caused by: >>>>>>>> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't >>>>>>>> deploy Yarn session cluster >>>>>>>> at >>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429) >>>>>>>> at >>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610) >>>>>>>> at >>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>>> at >>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) >>>>>>>> ... 2 more >>>>>>>> Caused by: >>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: >>>>>>>> The YARN application unexpectedly switched to state FAILED during >>>>>>>> deployment. >>>>>>>> Diagnostics from YARN: Application application_1584459865196_0165 >>>>>>>> failed 1 times (global limit =2; local limit is =1) due to AM >>>>>>>> Container for >>>>>>>> appattempt_1584459865196_0165_01 exited with exitCode: 1 >>>>>>>> Failing this attempt.Diagnostics: Exception from container-launch. >>>>>>>> Container id: container_1584459865196_0165_01_01 >>>>>>>> Exit code: 1 >>>>>>>> Exception message: Usage: java [-options] class [args...] >>>>>>>>(to execute a class) >>>>>>>>or java [-options] -jar jarfile [args...] >>>>>>>>(to execute a jar file) >>>>>>>> where options include: >>>>>>>> -d32 use a 32-bit data model if available >>>>>>>> -d64 use a 64-bit data model if available >>>>>>>> -server to select the "server" VM >>>>>>>> The default VM is server, >>>>>>>
Re: Flink Memory analyze on AWS EMR
on >>>>>>> emr when set up jitwatch configuration . >>>>>>> >>>>>>> 2020-05-11 19:06:09,552 ERROR >>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error >>>>>>> while >>>>>>> running the Flink Yarn session. >>>>>>> java.lang.reflect.UndeclaredThrowableException >>>>>>> at >>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) >>>>>>> at >>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>>>>>> at >>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813) >>>>>>> Caused by: >>>>>>> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't >>>>>>> deploy Yarn session cluster >>>>>>> at >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429) >>>>>>> at >>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610) >>>>>>> at >>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>> at >>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) >>>>>>> ... 2 more >>>>>>> Caused by: >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: >>>>>>> The YARN application unexpectedly switched to state FAILED during >>>>>>> deployment. >>>>>>> Diagnostics from YARN: Application application_1584459865196_0165 >>>>>>> failed 1 times (global limit =2; local limit is =1) due to AM Container >>>>>>> for >>>>>>> appattempt_1584459865196_0165_01 exited with exitCode: 1 >>>>>>> Failing this attempt.Diagnostics: Exception from container-launch. >>>>>>> Container id: container_1584459865196_0165_01_01 >>>>>>> Exit code: 1 >>>>>>> Exception message: Usage: java [-options] class [args...] >>>>>>>(to execute a class) >>>>>>>or java [-options] -jar jarfile [args...] >>>>>>>(to execute a jar file) >>>>>>> where options include: >>>>>>> -d32 use a 32-bit data model if available >>>>>>> -d64 use a 64-bit data model if available >>>>>>> -server to select the "server" VM >>>>>>> The default VM is server, >>>>>>> because you are running on a server-class machine. >>>>>>> >>>>>>> >>>>>>> -cp >>>>>>> -classpath >>>>>>> A : separated list of directories, JAR archives, >>>>>>> and ZIP archives to search for class files. >>>>>>> -D= >>>>>>> set a system property >>>>>>> -verbose:[class|gc|jni] >>>>>>> enable verbose output >>>>>>> -version print product version and exit >>>>>>> -version: >>>>>>> Warning: this feature is deprecated and will be >>>>>>> removed >>>>>>> in a future release. >>>>>>> require the specified version to run >>>>>>> -showversion print product version and continue >>>>>>> -jre-restrict-search | -no-jre-restrict-search >>>>>>> Warning: this feature is deprecated and will be >>>>>>> removed >>>>>>> in a future release. >>>>>>> include/exclude user private JREs in the version >>>>>>> search >>>>>>> -? -help print this help message >>>>>>> -Xprint help on non-standard options >>>>>>> -ea[:...|:] >>>>>>> -enableassertions[:...|:] >>>>>>> enable assertions with specified granularity >>>>>>> -da[:...|:] >>>>>>> -disableassertions[:...|:] >>>>>>> disable assertions with specified granularity >>>>>>> -esa | -enablesystemassertions >>>>>>> enable system assertions >>>>>>> -dsa | -disablesystemassertions >>>>>>> disable system assertions >>>>>>> -agentlib:[=] >>>>>>> load native agent library , e.g. >>>>>>> -agentlib:hprof >>>>>>> see also, -agentlib:jdwp=help and >>>>>>> -agentlib:hprof=help >>>>>>> -agentpath:[=] >>>>>>> load native agent library by full pathname >>>>>>> -javaagent:[=] >>>>>>> load Java programming language agent, see >>>>>>> java.lang.instrument >>>>>>> -splash: >>>>>>> show splash screen with specified image >>>>>>> See >>>>>>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>>>>>> for more details. >>>>>>> >>>>>>> Thanks >>>>>>> Jacky >>>>>>> >>>>>>> Robert Metzger 于2020年5月11日周一 下午3:42写道： >>>>>>> >>>>>>>> Hey Jacky, >>>>>>>> >>>>>>>> The error says "The YARN application unexpectedly switched to state >>>>>>>> FAILED during deployment.". >>>>>>>> Have you tried retrieving the YARN application logs? >>>>>>>> Does the YARN UI / resource manager logs reveal anything on the >>>>>>>> reason for the deployment to fail? >>>>>>>> >>>>>>>> Best, >>>>>>>> Robert >>>>>>>> >>>>>>>> >>>>>>>> On Mon, May 11, 2020 at 9:34 PM Jacky D >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- Forwarded message - >>>>>>>>> 发件人： Jacky D >>>>>>>>> Date: 2020年5月11日周一 下午3:12 >>>>>>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>>>>>> To: Khachatryan Roman >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, Roman >>>>>>>>> >>>>>>>>> Thanks for quick response , I tried without logFIle option but >>>>>>>>> failed with same error , I'm currently using flink 1.6 >>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>>>>>> so I can only use Jitwatch or JMC . I guess those tools only >>>>>>>>> available on >>>>>>>>> Standalone cluster ? as document mentioned "Each standalone >>>>>>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>>>>>> stdout and stderr to a file with a .out filename suffix and >>>>>>>>> writes internal logging to a file with a .log suffix. Java >>>>>>>>> options configured by the user in env.java.opts" ? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Jacky >>>>>>>>> >>>>>>>> >> >> -- >> >> Arvid Heise | Senior Java Developer >> >> <https://www.ververica.com/> >> >> Follow us @VervericaData >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Toni) Cheng >> >
Re: Flink Memory analyze on AWS EMR
avax.security.auth.Subject.doAs(Subject.java:422) >>>>>> at >>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) >>>>>> ... 2 more >>>>>> Caused by: >>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: >>>>>> The YARN application unexpectedly switched to state FAILED during >>>>>> deployment. >>>>>> Diagnostics from YARN: Application application_1584459865196_0165 >>>>>> failed 1 times (global limit =2; local limit is =1) due to AM Container >>>>>> for >>>>>> appattempt_1584459865196_0165_01 exited with exitCode: 1 >>>>>> Failing this attempt.Diagnostics: Exception from container-launch. >>>>>> Container id: container_1584459865196_0165_01_01 >>>>>> Exit code: 1 >>>>>> Exception message: Usage: java [-options] class [args...] >>>>>>(to execute a class) >>>>>>or java [-options] -jar jarfile [args...] >>>>>>(to execute a jar file) >>>>>> where options include: >>>>>> -d32 use a 32-bit data model if available >>>>>> -d64 use a 64-bit data model if available >>>>>> -server to select the "server" VM >>>>>> The default VM is server, >>>>>> because you are running on a server-class machine. >>>>>> >>>>>> >>>>>> -cp >>>>>> -classpath >>>>>> A : separated list of directories, JAR archives, >>>>>> and ZIP archives to search for class files. >>>>>> -D= >>>>>> set a system property >>>>>> -verbose:[class|gc|jni] >>>>>> enable verbose output >>>>>> -version print product version and exit >>>>>> -version: >>>>>> Warning: this feature is deprecated and will be >>>>>> removed >>>>>> in a future release. >>>>>> require the specified version to run >>>>>> -showversion print product version and continue >>>>>> -jre-restrict-search | -no-jre-restrict-search >>>>>> Warning: this feature is deprecated and will be >>>>>> removed >>>>>> in a future release. >>>>>> include/exclude user private JREs in the version >>>>>> search >>>>>> -? -help print this help message >>>>>> -Xprint help on non-standard options >>>>>> -ea[:...|:] >>>>>> -enableassertions[:...|:] >>>>>> enable assertions with specified granularity >>>>>> -da[:...|:] >>>>>> -disableassertions[:...|:] >>>>>> disable assertions with specified granularity >>>>>> -esa | -enablesystemassertions >>>>>> enable system assertions >>>>>> -dsa | -disablesystemassertions >>>>>> disable system assertions >>>>>> -agentlib:[=] >>>>>> load native agent library , e.g. >>>>>> -agentlib:hprof >>>>>> see also, -agentlib:jdwp=help and >>>>>> -agentlib:hprof=help >>>>>> -agentpath:[=] >>>>>> load native agent library by full pathname >>>>>> -javaagent:[=] >>>>>> load Java programming language agent, see >>>>>> java.lang.instrument >>>>>> -splash: >>>>>> show splash screen with specified image >>>>>> See >>>>>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>>>>> for more details. >>>>>> >>>>>> Thanks >>>>>> Jacky >>>>>> >>>>>> Robert Metzger 于2020年5月11日周一 下午3:42写道： >>>>>> >>>>>>> Hey Jacky, >>>>>>> >>>>>>> The error says "The YARN application unexpectedly switched to state >>>>>>> FAILED during deployment.". >>>>>>> Have you tried retrieving the YARN application logs? >>>>>>> Does the YARN UI / resource manager logs reveal anything on the >>>>>>> reason for the deployment to fail? >>>>>>> >>>>>>> Best, >>>>>>> Robert >>>>>>> >>>>>>> >>>>>>> On Mon, May 11, 2020 at 9:34 PM Jacky D >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- Forwarded message - >>>>>>>> 发件人： Jacky D >>>>>>>> Date: 2020年5月11日周一 下午3:12 >>>>>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>>>>> To: Khachatryan Roman >>>>>>>> >>>>>>>> >>>>>>>> Hi, Roman >>>>>>>> >>>>>>>> Thanks for quick response , I tried without logFIle option but >>>>>>>> failed with same error , I'm currently using flink 1.6 >>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>>>>> so I can only use Jitwatch or JMC . I guess those tools only >>>>>>>> available on >>>>>>>> Standalone cluster ? as document mentioned "Each standalone >>>>>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>>>>> stdout and stderr to a file with a .out filename suffix and writes >>>>>>>> internal logging to a file with a .log suffix. Java options >>>>>>>> configured by the user in env.java.opts" ? >>>>>>>> >>>>>>>> Thanks >>>>>>>> Jacky >>>>>>>> >>>>>>> > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
Re: Flink Memory analyze on AWS EMR
ata model if available >>>>> -server to select the "server" VM >>>>> The default VM is server, >>>>> because you are running on a server-class machine. >>>>> >>>>> >>>>> -cp >>>>> -classpath >>>>> A : separated list of directories, JAR archives, >>>>> and ZIP archives to search for class files. >>>>> -D= >>>>> set a system property >>>>> -verbose:[class|gc|jni] >>>>> enable verbose output >>>>> -version print product version and exit >>>>> -version: >>>>> Warning: this feature is deprecated and will be >>>>> removed >>>>> in a future release. >>>>> require the specified version to run >>>>> -showversion print product version and continue >>>>> -jre-restrict-search | -no-jre-restrict-search >>>>> Warning: this feature is deprecated and will be >>>>> removed >>>>> in a future release. >>>>> include/exclude user private JREs in the version >>>>> search >>>>> -? -help print this help message >>>>> -Xprint help on non-standard options >>>>> -ea[:...|:] >>>>> -enableassertions[:...|:] >>>>> enable assertions with specified granularity >>>>> -da[:...|:] >>>>> -disableassertions[:...|:] >>>>> disable assertions with specified granularity >>>>> -esa | -enablesystemassertions >>>>> enable system assertions >>>>> -dsa | -disablesystemassertions >>>>> disable system assertions >>>>> -agentlib:[=] >>>>> load native agent library , e.g. >>>>> -agentlib:hprof >>>>> see also, -agentlib:jdwp=help and >>>>> -agentlib:hprof=help >>>>> -agentpath:[=] >>>>> load native agent library by full pathname >>>>> -javaagent:[=] >>>>> load Java programming language agent, see >>>>> java.lang.instrument >>>>> -splash: >>>>> show splash screen with specified image >>>>> See >>>>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>>>> for more details. >>>>> >>>>> Thanks >>>>> Jacky >>>>> >>>>> Robert Metzger 于2020年5月11日周一 下午3:42写道： >>>>> >>>>>> Hey Jacky, >>>>>> >>>>>> The error says "The YARN application unexpectedly switched to state >>>>>> FAILED during deployment.". >>>>>> Have you tried retrieving the YARN application logs? >>>>>> Does the YARN UI / resource manager logs reveal anything on the >>>>>> reason for the deployment to fail? >>>>>> >>>>>> Best, >>>>>> Robert >>>>>> >>>>>> >>>>>> On Mon, May 11, 2020 at 9:34 PM Jacky D >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> -- Forwarded message - >>>>>>> 发件人： Jacky D >>>>>>> Date: 2020年5月11日周一 下午3:12 >>>>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>>>> To: Khachatryan Roman >>>>>>> >>>>>>> >>>>>>> Hi, Roman >>>>>>> >>>>>>> Thanks for quick response , I tried without logFIle option but >>>>>>> failed with same error , I'm currently using flink 1.6 >>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>>>> so I can only use Jitwatch or JMC . I guess those tools only available >>>>>>> on >>>>>>> Standalone cluster ? as document mentioned "Each standalone >>>>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>>>> stdout and stderr to a file with a .out filename suffix and writes >>>>>>> internal logging to a file with a .log suffix. Java options >>>>>>> configured by the user in env.java.opts" ? >>>>>>> >>>>>>> Thanks >>>>>>> Jacky >>>>>>> >>>>>> -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
Re: Flink Memory analyze on AWS EMR
-restrict-search | -no-jre-restrict-search >>>> Warning: this feature is deprecated and will be >>>> removed >>>> in a future release. >>>> include/exclude user private JREs in the version >>>> search >>>> -? -help print this help message >>>> -Xprint help on non-standard options >>>> -ea[:...|:] >>>> -enableassertions[:...|:] >>>> enable assertions with specified granularity >>>> -da[:...|:] >>>> -disableassertions[:...|:] >>>> disable assertions with specified granularity >>>> -esa | -enablesystemassertions >>>> enable system assertions >>>> -dsa | -disablesystemassertions >>>> disable system assertions >>>> -agentlib:[=] >>>> load native agent library , e.g. >>>> -agentlib:hprof >>>> see also, -agentlib:jdwp=help and -agentlib:hprof=help >>>> -agentpath:[=] >>>> load native agent library by full pathname >>>> -javaagent:[=] >>>> load Java programming language agent, see >>>> java.lang.instrument >>>> -splash: >>>> show splash screen with specified image >>>> See >>>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>>> for more details. >>>> >>>> Thanks >>>> Jacky >>>> >>>> Robert Metzger 于2020年5月11日周一 下午3:42写道： >>>> >>>>> Hey Jacky, >>>>> >>>>> The error says "The YARN application unexpectedly switched to state >>>>> FAILED during deployment.". >>>>> Have you tried retrieving the YARN application logs? >>>>> Does the YARN UI / resource manager logs reveal anything on the reason >>>>> for the deployment to fail? >>>>> >>>>> Best, >>>>> Robert >>>>> >>>>> >>>>> On Mon, May 11, 2020 at 9:34 PM Jacky D >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> -- Forwarded message - >>>>>> 发件人： Jacky D >>>>>> Date: 2020年5月11日周一 下午3:12 >>>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>>> To: Khachatryan Roman >>>>>> >>>>>> >>>>>> Hi, Roman >>>>>> >>>>>> Thanks for quick response , I tried without logFIle option but failed >>>>>> with same error , I'm currently using flink 1.6 >>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>>> so I can only use Jitwatch or JMC . I guess those tools only available >>>>>> on >>>>>> Standalone cluster ? as document mentioned "Each standalone >>>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>>> stdout and stderr to a file with a .out filename suffix and writes >>>>>> internal logging to a file with a .log suffix. Java options >>>>>> configured by the user in env.java.opts" ? >>>>>> >>>>>> Thanks >>>>>> Jacky >>>>>> >>>>>
Re: Flink Memory analyze on AWS EMR
;> in a future release. >>> include/exclude user private JREs in the version search >>> -? -help print this help message >>> -Xprint help on non-standard options >>> -ea[:...|:] >>> -enableassertions[:...|:] >>> enable assertions with specified granularity >>> -da[:...|:] >>> -disableassertions[:...|:] >>> disable assertions with specified granularity >>> -esa | -enablesystemassertions >>> enable system assertions >>> -dsa | -disablesystemassertions >>> disable system assertions >>> -agentlib:[=] >>> load native agent library , e.g. >>> -agentlib:hprof >>> see also, -agentlib:jdwp=help and -agentlib:hprof=help >>> -agentpath:[=] >>> load native agent library by full pathname >>> -javaagent:[=] >>> load Java programming language agent, see >>> java.lang.instrument >>> -splash: >>> show splash screen with specified image >>> See >>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>> for more details. >>> >>> Thanks >>> Jacky >>> >>> Robert Metzger 于2020年5月11日周一 下午3:42写道： >>> >>>> Hey Jacky, >>>> >>>> The error says "The YARN application unexpectedly switched to state >>>> FAILED during deployment.". >>>> Have you tried retrieving the YARN application logs? >>>> Does the YARN UI / resource manager logs reveal anything on the reason >>>> for the deployment to fail? >>>> >>>> Best, >>>> Robert >>>> >>>> >>>> On Mon, May 11, 2020 at 9:34 PM Jacky D wrote: >>>> >>>>> >>>>> >>>>> -- Forwarded message - >>>>> 发件人： Jacky D >>>>> Date: 2020年5月11日周一 下午3:12 >>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>> To: Khachatryan Roman >>>>> >>>>> >>>>> Hi, Roman >>>>> >>>>> Thanks for quick response , I tried without logFIle option but failed >>>>> with same error , I'm currently using flink 1.6 >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>> so I can only use Jitwatch or JMC . I guess those tools only available on >>>>> Standalone cluster ? as document mentioned "Each standalone >>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>> stdout and stderr to a file with a .out filename suffix and writes >>>>> internal logging to a file with a .log suffix. Java options >>>>> configured by the user in env.java.opts" ? >>>>> >>>>> Thanks >>>>> Jacky >>>>> >>>>
Re: Flink Memory analyze on AWS EMR
ble system assertions >> -agentlib:[=] >> load native agent library , e.g. >> -agentlib:hprof >> see also, -agentlib:jdwp=help and -agentlib:hprof=help >> -agentpath:[=] >> load native agent library by full pathname >> -javaagent:[=] >> load Java programming language agent, see >> java.lang.instrument >> -splash: >> show splash screen with specified image >> See >> http://www.oracle.com/technetwork/java/javase/documentation/index.html >> for more details. >> >> Thanks >> Jacky >> >> Robert Metzger 于2020年5月11日周一 下午3:42写道： >> >>> Hey Jacky, >>> >>> The error says "The YARN application unexpectedly switched to state >>> FAILED during deployment.". >>> Have you tried retrieving the YARN application logs? >>> Does the YARN UI / resource manager logs reveal anything on the reason >>> for the deployment to fail? >>> >>> Best, >>> Robert >>> >>> >>> On Mon, May 11, 2020 at 9:34 PM Jacky D wrote: >>> >>>> >>>> >>>> -- Forwarded message - >>>> 发件人： Jacky D >>>> Date: 2020年5月11日周一 下午3:12 >>>> Subject: Re: Flink Memory analyze on AWS EMR >>>> To: Khachatryan Roman >>>> >>>> >>>> Hi, Roman >>>> >>>> Thanks for quick response , I tried without logFIle option but failed >>>> with same error , I'm currently using flink 1.6 >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>> so I can only use Jitwatch or JMC . I guess those tools only available on >>>> Standalone cluster ? as document mentioned "Each standalone >>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>> stdout and stderr to a file with a .out filename suffix and writes >>>> internal logging to a file with a .log suffix. Java options configured >>>> by the user in env.java.opts" ? >>>> >>>> Thanks >>>> Jacky >>>> >>> memErrorLog.log Description: Binary data
Re: Flink Memory analyze on AWS EMR
x.html > for more details. > > Thanks > Jacky > > Robert Metzger 于2020年5月11日周一 下午3:42写道： > >> Hey Jacky, >> >> The error says "The YARN application unexpectedly switched to state >> FAILED during deployment.". >> Have you tried retrieving the YARN application logs? >> Does the YARN UI / resource manager logs reveal anything on the reason >> for the deployment to fail? >> >> Best, >> Robert >> >> >> On Mon, May 11, 2020 at 9:34 PM Jacky D wrote: >> >>> >>> >>> -- Forwarded message - >>> 发件人： Jacky D >>> Date: 2020年5月11日周一 下午3:12 >>> Subject: Re: Flink Memory analyze on AWS EMR >>> To: Khachatryan Roman >>> >>> >>> Hi, Roman >>> >>> Thanks for quick response , I tried without logFIle option but failed >>> with same error , I'm currently using flink 1.6 >>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>> so I can only use Jitwatch or JMC . I guess those tools only available on >>> Standalone cluster ? as document mentioned "Each standalone JobManager, >>> TaskManager, HistoryServer, and ZooKeeper daemon redirects stdout and >>> stderr to a file with a .out filename suffix and writes internal >>> logging to a file with a .log suffix. Java options configured by the >>> user in env.java.opts" ? >>> >>> Thanks >>> Jacky >>> >>
Re: Flink Memory analyze on AWS EMR
Hi，Robert Yes , I tried to retrieve more log info from yarn UI , the full logs showing below , this happens when I try to create a flink yarn session on emr when set up jitwatch configuration . 2020-05-11 19:06:09,552 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while running the Flink Yarn session. java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813) Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) ... 2 more Caused by: org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. Diagnostics from YARN: Application application_1584459865196_0165 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1584459865196_0165_01 exited with exitCode: 1 Failing this attempt.Diagnostics: Exception from container-launch. Container id: container_1584459865196_0165_01_01 Exit code: 1 Exception message: Usage: java [-options] class [args...] (to execute a class) or java [-options] -jar jarfile [args...] (to execute a jar file) where options include: -d32 use a 32-bit data model if available -d64 use a 64-bit data model if available -server to select the "server" VM The default VM is server, because you are running on a server-class machine. -cp -classpath A : separated list of directories, JAR archives, and ZIP archives to search for class files. -D= set a system property -verbose:[class|gc|jni] enable verbose output -version print product version and exit -version: Warning: this feature is deprecated and will be removed in a future release. require the specified version to run -showversion print product version and continue -jre-restrict-search | -no-jre-restrict-search Warning: this feature is deprecated and will be removed in a future release. include/exclude user private JREs in the version search -? -help print this help message -Xprint help on non-standard options -ea[:...|:] -enableassertions[:...|:] enable assertions with specified granularity -da[:...|:] -disableassertions[:...|:] disable assertions with specified granularity -esa | -enablesystemassertions enable system assertions -dsa | -disablesystemassertions disable system assertions -agentlib:[=] load native agent library , e.g. -agentlib:hprof see also, -agentlib:jdwp=help and -agentlib:hprof=help -agentpath:[=] load native agent library by full pathname -javaagent:[=] load Java programming language agent, see java.lang.instrument -splash: show splash screen with specified image See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details. Thanks Jacky Robert Metzger 于2020年5月11日周一 下午3:42写道： > Hey Jacky, > > The error says "The YARN application unexpectedly switched to state FAILED > during deployment.". > Have you tried retrieving the YARN application logs? > Does the YARN UI / resource manager logs reveal anything on the reason for > the deployment to fail? > > Best, > Robert > > > On Mon, May 11, 2020 at 9:34 PM Jacky D wrote: > >> >> >> -- Forwarded message - >> 发件人： Jacky D >> Date: 2020年5月11日周一 下午3:12 >> Subject: Re: Flink Memory analyze on AWS EMR >> To: Khachatryan Roman >> >> >> Hi, Roman >> >> Thanks for quick response , I tried without logFIle option but failed >> with same error , I'm currently using flink 1.6 >> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >> so I can only use Jitwatch
Re: Flink Memory analyze on AWS EMR
Hey Jacky, The error says "The YARN application unexpectedly switched to state FAILED during deployment.". Have you tried retrieving the YARN application logs? Does the YARN UI / resource manager logs reveal anything on the reason for the deployment to fail? Best, Robert On Mon, May 11, 2020 at 9:34 PM Jacky D wrote: > > > -- Forwarded message - > 发件人： Jacky D > Date: 2020年5月11日周一 下午3:12 > Subject: Re: Flink Memory analyze on AWS EMR > To: Khachatryan Roman > > > Hi, Roman > > Thanks for quick response , I tried without logFIle option but failed with > same error , I'm currently using flink 1.6 > https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, > so I can only use Jitwatch or JMC . I guess those tools only available on > Standalone cluster ? as document mentioned "Each standalone JobManager, > TaskManager, HistoryServer, and ZooKeeper daemon redirects stdout and > stderr to a file with a .out filename suffix and writes internal logging > to a file with a .log suffix. Java options configured by the user in > env.java.opts" ? > > Thanks > Jacky >
Fwd: Flink Memory analyze on AWS EMR
-- Forwarded message - 发件人： Jacky D Date: 2020年5月11日周一 下午3:12 Subject: Re: Flink Memory analyze on AWS EMR To: Khachatryan Roman Hi, Roman Thanks for quick response , I tried without logFIle option but failed with same error , I'm currently using flink 1.6 https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, so I can only use Jitwatch or JMC . I guess those tools only available on Standalone cluster ? as document mentioned "Each standalone JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects stdout and stderr to a file with a .out filename suffix and writes internal logging to a file with a .log suffix. Java options configured by the user in env.java.opts" ? Thanks Jacky
Re: Flink Memory analyze on AWS EMR
Hi Jacky,
Did you try it without -XX:LogFile=${FLINK_LOG_PREFIX}.jit ?
Probably, Flink can't write to this location.
Also, you can try other tools described at
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/application_profiling.html
Regards,
Roman
On Mon, May 11, 2020 at 5:02 PM Jacky D wrote:
> hi, All
>
> I'm encounter a memory issue with my flink job on AWS EMR(current flink
> version 1.6.2) , I would like to find the root cause so I'm trying JITWatch
> on my local standalone cluster but I can not use it on EMR . after I add
> following config on flink-conf.yaml :
>
> env.java.opts: "-XX:+UnlockDiagnosticVMOptions -XX:+TraceClassLoading
> -XX:+LogCompilation -XX:LogFile=${FLINK_LOG_PREFIX}.jit -XX:+PrintAssembly"
>
> I got error
>
> 2020-05-07 16:24:53,368 ERROR
> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while
> running the Flink Yarn session.
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813)
> Caused by: org.apache.flink.client.deployment.ClusterDeploymentException:
> Couldn't deploy Yarn session cluster
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> ... 2 more
> Caused by:
> org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException:
> The YARN application unexpectedly switched to state FAILED during
> deployment.
>
> How can I fix this issue to enable JITWatch or which tool will be a proper
> way to analyze flink mem dump on EMR ?
>
> Thanks
> Jacky Du
>
Flink Memory analyze on AWS EMR
hi, All
I'm encounter a memory issue with my flink job on AWS EMR(current flink
version 1.6.2) , I would like to find the root cause so I'm trying JITWatch
on my local standalone cluster but I can not use it on EMR . after I add
following config on flink-conf.yaml :
env.java.opts: "-XX:+UnlockDiagnosticVMOptions -XX:+TraceClassLoading
-XX:+LogCompilation -XX:LogFile=${FLINK_LOG_PREFIX}.jit -XX:+PrintAssembly"
I got error
2020-05-07 16:24:53,368 ERROR
org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while
running the Flink Yarn session.
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813)
Caused by: org.apache.flink.client.deployment.ClusterDeploymentException:
Couldn't deploy Yarn session cluster
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
... 2 more
Caused by:
org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException:
The YARN application unexpectedly switched to state FAILED during
deployment.
How can I fix this issue to enable JITWatch or which tool will be a proper
way to analyze flink mem dump on EMR ?
Thanks
Jacky Du
