[
https://issues.apache.org/jira/browse/FLINK-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288150#comment-17288150
]
hayden zhou commented on FLINK-21325:
-------------------------------------
i found some exception like this
{code:java}
2021-02-22 11:07:05,150 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
TableSourceScan(table=[[default_catalog, default_database, rowDataTable,
project=[app_id, ip, ts, action]]], fields=[app_id, ip, ts, action]) ->
Calc(select=[app_id, ip], where=[((ts SEARCH Sarg[[1613955600..1613959200)])
AND (action SEARCH Sarg[_UTF-16LE'PV':VARCHAR(2147483647) CHARACTER SET
"UTF-16LE"]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))]) ->
Expand(projects=[app_id, ip, $e], projects=[{app_id, ip, 0 AS $e}, {app_id,
null AS ip, 1 AS $e}]) -> LocalHashAggregate(groupBy=[app_id, ip, $e],
select=[app_id, ip, $e, Partial_COUNT(*) AS count1$0]) (1/1)
(c72791dbe1987c48cb4c302bfe422a62) switched from CREATED to SCHEDULED.
2021-02-22 11:07:05,150 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve
slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{e93cb76a2cf4a64e5498ecb9be4d93d0}]
2021-02-22 11:07:05,151 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
TableSourceScan(table=[[default_catalog, default_database, rowDataTable,
project=[app_id, path, ts, action, uid]]], fields=[app_id, path, ts, action,
uid]) -> Calc(select=[app_id, uid, path], where=[((ts SEARCH
Sarg[[1613955600..1613959200)]) AND (action SEARCH
Sarg[_UTF-16LE'PV':VARCHAR(2147483647) CHARACTER SET
"UTF-16LE"]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))]) ->
LocalHashAggregate(groupBy=[app_id, uid, path], select=[app_id, uid, path])
(1/1) (4b57afd56e9543727b72ad6b86e088d1) switched from CREATED to SCHEDULED.
2021-02-22 11:07:05,151 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve
slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{05243bdf664021ad623b0714e97c9f9d}]
2021-02-22 11:07:05,151 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
TableSourceScan(table=[[default_catalog, default_database, rowDataTable,
project=[app_id, ts, action, dac, uid]]], fields=[app_id, ts, action, dac,
uid]) -> Calc(select=[app_id, CAST(dac) AS $f1, uid], where=[((ts SEARCH
Sarg[[1613955600..1613959200)]) AND (action SEARCH
Sarg[_UTF-16LE'HB':VARCHAR(2147483647) CHARACTER SET
"UTF-16LE"]:VARCHAR(2147483647) CHARACTER SET "UTF-16LE"))]) ->
Expand(projects=[app_id, $f1, uid, $e], projects=[{app_id, $f1, uid, 0 AS $e},
{app_id, $f1, null AS uid, 1 AS $e}]) -> LocalHashAggregate(groupBy=[app_id,
uid, $e], select=[app_id, uid, $e, Partial_SUM($f1) AS sum$0]) (1/1)
(953ab9777f59696592b246613c519a8e) switched from CREATED to SCHEDULED.
2021-02-22 11:07:05,151 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Cannot serve
slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{e4425f7443819b5db22f0204db200431}]
{code}
> NoResourceAvailableException while cancel then resubmit jobs
> -------------------------------------------------------------
>
> Key: FLINK-21325
> URL: https://issues.apache.org/jira/browse/FLINK-21325
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Coordination
> Environment: FLINK 1.12 with
> [flink-kubernetes_2.11-1.12-SNAPSHOT.jar] in libs directory to fix FLINK
> restart problem on k8s HA session mode.
> Reporter: hayden zhou
> Priority: Major
>
> I have five stream jobs and want to clear all states in jobs, so I canceled
> all those jobs, then resubmitted one by one, resulting in two jobs are in
> running status, while three jobs are in created status with errors
> ".NoResourceAvailableException: Slot request bulk is not fulfillable! Could
> not allocate the required slot within slot request timeout
> "
> I am sure my slots are sufficient.
> and this problem were fixed by restart k8s jm an tm pods.
> below is the error logs:
> {code:java}
> ava.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at
> org.apache.flink.runtime.scheduler.SharedSlot.cancelLogicalSlotRequest(SharedSlot.java:195)
> at
> org.apache.flink.runtime.scheduler.SlotSharingExecutionSlotAllocator.cancelLogicalSlotRequest(SlotSharingExecutionSlotAllocator.java:147)
> at
> org.apache.flink.runtime.scheduler.SharingPhysicalSlotRequestBulk.cancel(SharingPhysicalSlotRequestBulk.java:84)
> at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkWithTimestamp.cancel(PhysicalSlotRequestBulkWithTimestamp.java:66)
> at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:87)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:404)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:197)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
> at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:84)
> ... 24 more
> Caused by: java.util.concurrent.TimeoutException: Timeout has occurred:
> 300000 ms
> ... 25 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
