[
https://issues.apache.org/jira/browse/FLINK-36291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882408#comment-17882408
]
Matthias Pohl commented on FLINK-36291:
---------------------------------------
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=62064&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=23909
causing some side noise:
{code}
Sep 13 03:13:49 03:13:50.002 [ERROR] Tests run: 22, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 509.6 s <<< FAILURE! -- in
org.apache.flink.connectors.hive.HiveTableSourceITCase
Sep 13 03:13:50 03:13:50.002 [ERROR]
org.apache.flink.connectors.hive.HiveTableSourceITCase.testNonPartitionStreamingSourceWithVectorizedReader
-- Time elapsed: 30.06 s <<< ERROR!
Sep 13 03:13:50 org.junit.runners.model.TestTimedOutException: test timed out
after 30000 milliseconds
Sep 13 03:13:50 at sun.misc.Unsafe.unpark(Native Method)
[...]
{code}
and
{code}
Sep 13 04:07:35 at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
Sep 13 04:07:35 at
org.apache.pekko.pattern.PromiseActorRef.$bang(AskSupport.scala:629)
Sep 13 04:07:35 at
org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:34)
Sep 13 04:07:35 at
org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:33)
Sep 13 04:07:35 at
scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536)
Sep 13 04:07:35 at
scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
Sep 13 04:07:35 at
scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
Sep 13 04:07:35 at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
Sep 13 04:07:35 at
org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:73)
Sep 13 04:07:35 at
org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:110)
Sep 13 04:07:35 at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
Sep 13 04:07:35 at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
Sep 13 04:07:35 at
org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:110)
Sep 13 04:07:35 at
org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
Sep 13 04:07:35 at
org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57)
Sep 13 04:07:35 ... 4 more
Sep 13 04:07:35 Caused by: org.apache.flink.runtime.JobException: Recovery is
suppressed by NoRestartBackoffTimeStrategy
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:281)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:272)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler.handleTaskFailure(AdaptiveBatchScheduler.java:413)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:265)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler.onTaskFailed(AdaptiveBatchScheduler.java:405)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:800)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:777)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:51)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(DefaultExecutionGraph.java:1675)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1190)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1130)
Sep 13 04:07:35 at
org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:831)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.signalPayloadRelease(SingleLogicalSlot.java:195)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot.release(SingleLogicalSlot.java:182)
Sep 13 04:07:35 at
org.apache.flink.runtime.scheduler.SimpleExecutionSlotAllocator$LogicalSlotHolder.release(SimpleExecutionSlotAllocator.java:203)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:152)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool.releasePayload(DefaultDeclarativeSlotPool.java:515)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool.freeAndReleaseSlots(DefaultDeclarativeSlotPool.java:507)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool.releaseSlots(DefaultDeclarativeSlotPool.java:478)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolService.internalReleaseTaskManager(DeclarativeSlotPoolService.java:281)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolService.releaseTaskManager(DeclarativeSlotPoolService.java:237)
Sep 13 04:07:35 at
org.apache.flink.runtime.jobmaster.JobMaster.disconnectTaskManager(JobMaster.java:571)
Sep 13 04:07:35 at java.lang.reflect.Method.invoke(Method.java:498)
[...]
Sep 13 04:07:35 ... 4 more
Sep 13 04:07:35 Caused by:
org.apache.flink.runtime.taskexecutor.exceptions.TaskManagerException: The
heartbeat of ResourceManager with id 75ce18d4ca9b983a6ce633d9787a8a2b timed out.
Sep 13 04:07:35 at
org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerHeartbeatListener.notifyHeartbeatTimeout(TaskExecutor.java:2783)
Sep 13 04:07:35 at
org.apache.flink.runtime.heartbeat.DefaultHeartbeatMonitor.run(DefaultHeartbeatMonitor.java:158)
Sep 13 04:07:35 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Sep 13 04:07:35 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Sep 13 04:07:35 at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRunAsync$4(PekkoRpcActor.java:460)
Sep 13 04:07:35 at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
Sep 13 04:07:35 at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRunAsync(PekkoRpcActor.java:460)
Sep 13 04:07:35 at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:225)
Sep 13 04:07:35 ... 21 more
{code}
> java.lang.IllegalMonitorStateException causing a fatal error on the
> TaskManager side
> ------------------------------------------------------------------------------------
>
> Key: FLINK-36291
> URL: https://issues.apache.org/jira/browse/FLINK-36291
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.0-preview
> Reporter: Matthias Pohl
> Priority: Blocker
>
> HiveDynamicPartitionPruningITCase failed due to the TM timeout. Checking the
> logs though revealed a fatal error on the taskmanager's side:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=62173&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=24046
> {code}
> 03:18:32,209 [taskmanager_72-main-scheduler-thread-1] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'taskmanager_72-main-scheduler-thread-1' produced an uncaught
> exception. Stopping the process...
> java.lang.IllegalMonitorStateException: null
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(AbstractQueuedSynchronizer.java:1939)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1103)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> {code}
> But there's also a OutOfMemoryError reported just a line below:
> {code}
> 03:19:01,060 [Source Data Fetcher for Source: part[62] (2/2)#0] ERROR
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager []
> - Received uncaught exception.
> java.lang.OutOfMemoryError: Java heap space
> {code}
> So that might be related to FLINK-36290
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
