[
https://issues.apache.org/jira/browse/FLINK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304696#comment-17304696
]
Dawid Wysakowicz commented on FLINK-21866:
------------------------------------------
cc [~trohrmann] [~rmetzger]
> JMXJobManagerMetricTest crashes on azure
> ----------------------------------------
>
> Key: FLINK-21866
> URL: https://issues.apache.org/jira/browse/FLINK-21866
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.13.0
> Reporter: Dawid Wysakowicz
> Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14990&view=logs&j=c91190b6-40ae-57b2-5999-31b869b0a7c1&t=43529380-51b4-5e90-5af4-2dccec0ef402&l=14134
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on
> project flink-metrics-jmx_2.11: There are test failures.
> [ERROR]
> [ERROR] Please refer to
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire-reports for the
> individual test results.
> [ERROR] Please refer to dump files (if any exist) [date].dump,
> [date]-jvmRun[N].dump and [date].dumpstream.
> [ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /__w/3/s/flink-metrics/flink-metrics-jmx &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire/surefirebooter6251600074636448230.jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire
> 2021-03-18T23-30-30_066-jvmRun2 surefire5926299627725589334tmp
> surefire_1185286162062283237015tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 239
> [ERROR] Crashed tests:
> [ERROR] org.apache.flink.runtime.jobmanager.JMXJobManagerMetricTest
> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /__w/3/s/flink-metrics/flink-metrics-jmx &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire/surefirebooter6251600074636448230.jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire
> 2021-03-18T23-30-30_066-jvmRun2 surefire5926299627725589334tmp
> surefire_1185286162062283237015tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 239
> [ERROR] Crashed tests:
> [ERROR] org.apache.flink.runtime.jobmanager.JMXJobManagerMetricTest
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:382)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /__w/3/s/flink-metrics/flink-metrics-jmx &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire/surefirebooter6251600074636448230.jar
> /__w/3/s/flink-metrics/flink-metrics-jmx/target/surefire
> 2021-03-18T23-30-30_066-jvmRun2 surefire5926299627725589334tmp
> surefire_1185286162062283237015tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 239
> [ERROR] Crashed tests:
> [ERROR] org.apache.flink.runtime.jobmanager.JMXJobManagerMetricTest
> [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
> [ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [ERROR] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [ERROR] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [ERROR] at java.lang.Thread.run(Thread.java:748)
> {code}
> Going through the logs I found out logs like the following:
> {code}
> 23:52:01,173 [flink-akka.actor.default-dispatcher-2] ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'flink-akka.actor.default-dispatcher-2' produced an uncaught
> exception. Stopping the process...
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkException: Could not properly shut down the
> JobManagerRunner
> at
> java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:998)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
> ~[?:1.8.0_282]
> at
> org.apache.flink.runtime.dispatcher.DispatcherJob.lambda$closeAsync$8(DispatcherJob.java:273)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:848)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2168)
> ~[?:1.8.0_282]
> at
> org.apache.flink.runtime.dispatcher.DispatcherJob.closeAsync(DispatcherJob.java:268)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.removeJob(Dispatcher.java:754)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$4(Dispatcher.java:432)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:814)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> ~[?:1.8.0_282]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> [scala-library-2.11.12.jar:?]
> at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> [scala-library-2.11.12.jar:?]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [scala-library-2.11.12.jar:?]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> [scala-library-2.11.12.jar:?]
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> [akka-actor_2.11-2.5.21.jar:2.5.21]
> Caused by: org.apache.flink.util.FlinkException: Could not properly shut down
> the JobManagerRunner
> at
> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.lambda$closeAsync$0(JobManagerRunnerImpl.java:224)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
> ~[?:1.8.0_282]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
> ~[?:1.8.0_282]
> at
> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.closeAsync(JobManagerRunnerImpl.java:208)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
> ~[?:1.8.0_282]
> ... 36 more
> Caused by: org.apache.flink.runtime.rpc.akka.exceptions.AkkaRpcException:
> Could not start RpcEndpoint jobmanager_3.
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:610)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.12.jar:?]
> at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> ~[scala-library-2.11.12.jar:?]
> ... 12 more
> Caused by: org.apache.flink.runtime.jobmaster.JobMasterException: Could not
> start the JobMaster.
> at
> org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:384)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.12.jar:?]
> at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> ~[scala-library-2.11.12.jar:?]
> ... 12 more
> Caused by: java.lang.IllegalArgumentException
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:122)
> ~[flink-core-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.util.ResourceCounter.withResource(ResourceCounter.java:244)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.scheduler.adaptive.allocator.SlotSharingSlotAllocator.calculateRequiredSlots(SlotSharingSlotAllocator.java:61)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.calculateDesiredResources(AdaptiveScheduler.java:728)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.goToWaitingForResources(AdaptiveScheduler.java:720)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.scheduler.adaptive.Created.startScheduling(Created.java:72)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.startScheduling(AdaptiveScheduler.java:308)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:960)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:382)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180)
> ~[flink-runtime_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.12.jar:?]
> at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> ~[akka-actor_2.11-2.5.21.jar:2.5.21]
> at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> ~[scala-library-2.11.12.jar:?]
> ... 12 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
