[
https://issues.apache.org/jira/browse/FLINK-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382038#comment-17382038
]
Till Rohrmann edited comment on FLINK-23409 at 7/16/21, 12:28 PM:
------------------------------------------------------------------
Hmm, locally everything passes. Also the build for FLINK-23093 passed.
What I found in the logs is the following:
{code}
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(2/4) (61b9f281e369b96551f4f42d7bc6b156) switched from CREATED to SCHEDULED.
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(3/4) (4d49ea1e8acbc3f8368ece58928e6c5f) switched from CREATED to SCHEDULED.
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(4/4) (e0e327a62ffe2733022a92cf4c605921) switched from CREATED to SCHEDULED.
09:16:36,419 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to
ResourceManager
akka://flink/user/rpc/resourcemanager_2(b2ae4decd3b74394548861ada12c402f)
09:16:36,420 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved
ResourceManager address, beginning registration
09:16:36,421 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registering job manager
ba78e82641830967d1cf39d623934f97@akka://flink/user/rpc/jobmanager_7 for job
72e507bcdef499061e756475aa153b19.
09:16:36,421 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registered job manager
ba78e82641830967d1cf39d623934f97@akka://flink/user/rpc/jobmanager_7 for job
72e507bcdef499061e756475aa153b19.
09:16:36,422 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager
successfully registered at ResourceManager, leader id:
b2ae4decd3b74394548861ada12c402f.
09:16:36,422 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Received resource requirements from job 72e507bcdef499061e756475aa153b19:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=4}]
09:17:33,474 [flink-akka.actor.default-dispatcher-11] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:18:03,493 [flink-akka.actor.default-dispatcher-22] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:18:33,512 [flink-akka.actor.default-dispatcher-28] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:19:03,532 [flink-akka.actor.default-dispatcher-34] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:19:33,552 [flink-akka.actor.default-dispatcher-37] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:20:03,587 [flink-akka.actor.default-dispatcher-45] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:20:33,601 [flink-akka.actor.default-dispatcher-51] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:03,613 [flink-akka.actor.default-dispatcher-56] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:33,632 [flink-akka.actor.default-dispatcher-64] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:36,443 [flink-akka.actor.default-dispatcher-65] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSource
(at
org.apache.flink.api.scala.util.CollectionDataSets$.getSmall5TupleDataSet(CollectionDataSets.scala:94)
(org.apache.flink.api.java.io.CollectionInpu) (1/1)
(9e7f450b1c1d7ef21005a7f237f8aae7) switched from SCHEDULED to FAILED on
[unassigned resource].
java.util.concurrent.CompletionException:
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Slot request bulk is not fulfillable! Could not allocate the required slot
within slot request timeout
{code}
Note that there are quite some gaps in the timestamps. This could point towards
some infrastructure problems.
Also, the build runs with fine grained resource management. Maybe this is
affecting the test case. cc [~xintongsong].
was (Author: till.rohrmann):
Hmm, locally everything passes. Also the build for FLINK-23093 passed.
What I found in the logs is the following:
{code}
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(2/4) (61b9f281e369b96551f4f42d7bc6b156) switched from CREATED to SCHEDULED.
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(3/4) (4d49ea1e8acbc3f8368ece58928e6c5f) switched from CREATED to SCHEDULED.
09:16:36,416 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSink
(CsvOutputFormat (path:
file:/tmp/junit7235602407204969305/junit837890199545668434.tmp, delimiter: ,))
(4/4) (e0e327a62ffe2733022a92cf4c605921) switched from CREATED to SCHEDULED.
09:16:36,419 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to
ResourceManager
akka://flink/user/rpc/resourcemanager_2(b2ae4decd3b74394548861ada12c402f)
09:16:36,420 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved
ResourceManager address, beginning registration
09:16:36,421 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registering job manager
ba78e82641830967d1cf39d623934f97@akka://flink/user/rpc/jobmanager_7 for job
72e507bcdef499061e756475aa153b19.
09:16:36,421 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
Registered job manager
ba78e82641830967d1cf39d623934f97@akka://flink/user/rpc/jobmanager_7 for job
72e507bcdef499061e756475aa153b19.
09:16:36,422 [flink-akka.actor.default-dispatcher-4] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager
successfully registered at ResourceManager, leader id:
b2ae4decd3b74394548861ada12c402f.
09:16:36,422 [flink-akka.actor.default-dispatcher-6] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Received resource requirements from job 72e507bcdef499061e756475aa153b19:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=4}]
09:17:33,474 [flink-akka.actor.default-dispatcher-11] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:18:03,493 [flink-akka.actor.default-dispatcher-22] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:18:33,512 [flink-akka.actor.default-dispatcher-28] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:19:03,532 [flink-akka.actor.default-dispatcher-34] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:19:33,552 [flink-akka.actor.default-dispatcher-37] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:20:03,587 [flink-akka.actor.default-dispatcher-45] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:20:33,601 [flink-akka.actor.default-dispatcher-51] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:03,613 [flink-akka.actor.default-dispatcher-56] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:33,632 [flink-akka.actor.default-dispatcher-64] INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Release TaskManager a61dbdd440a00d4a7dd5d7e16944f57f because it exceeded the
idle timeout.
09:21:36,443 [flink-akka.actor.default-dispatcher-65] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DataSource
(at
org.apache.flink.api.scala.util.CollectionDataSets$.getSmall5TupleDataSet(CollectionDataSets.scala:94)
(org.apache.flink.api.java.io.CollectionInpu) (1/1)
(9e7f450b1c1d7ef21005a7f237f8aae7) switched from SCHEDULED to FAILED on
[unassigned resource].
java.util.concurrent.CompletionException:
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Slot request bulk is not fulfillable! Could not allocate the required slot
within slot request timeout
{code}
Note that there are quite some gaps in the timestamps. Also, the build runs
with fine grained resource management. Maybe this is affecting it
[~xintongsong].
> CrossITCase fails with "NoResourceAvailableException: Slot request bulk is
> not fulfillable! Could not allocate the required slot within slot request
> timeout"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-23409
> URL: https://issues.apache.org/jira/browse/FLINK-23409
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Table SQL / Planner
> Affects Versions: 1.14.0
> Reporter: Dawid Wysakowicz
> Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20548&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=5360d54c-8d94-5d85-304e-a89267eb785a&l=10074
> {code}
> Jul 16 09:21:37 at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> Jul 16 09:21:37 at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> Jul 16 09:21:37 at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> Jul 16 09:21:37 at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> Jul 16 09:21:37 at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> Jul 16 09:21:37 at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> Jul 16 09:21:37 at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> Jul 16 09:21:37 at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> Jul 16 09:21:37 ... 4 more
> Jul 16 09:21:37 Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
> Jul 16 09:21:37 at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> Jul 16 09:21:37 at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> Jul 16 09:21:37 at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> Jul 16 09:21:37 at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> Jul 16 09:21:37 ... 31 more
> Jul 16 09:21:37 Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
> Jul 16 09:21:37 at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
> Jul 16 09:21:37 ... 24 more
> Jul 16 09:21:37 Caused by: java.util.concurrent.TimeoutException: Timeout has
> occurred: 300000 ms
> Jul 16 09:21:37 ... 25 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
