[
https://issues.apache.org/jira/browse/FLINK-40789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137024#comment-18137024
]
Naveen Joseph commented on FLINK-40789:
---------------------------------------
[~martijnvisser], the test cases themselves look fine. However, the issue
appears to be related to resource availability and resource handling, as these
failures occur only on Azure Pipeline agents.
Initially, I reproduced this issue under stress conditions. I then checked our
pipeline wiki for information about the underlying infrastructure. According to
the documentation, we have four VMs, each with 32 cores and 64 GB of memory,
running two agents per VM. At first glance, this seems like a sufficient amount
of resources for running Flink builds and tests.
However, after investigating the jobs in [Azure Pipeline run
#20261025.1|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=80472&view=logs&si=ae4f8708-9994-57d3-c2d7-b892156e7812]
and run
#[#[20261001.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=79631&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=1ffc5ec2-7913-50ff-0177-3fca16f1b8f0]],
I noticed that more than two agents appear to be running on the same VM
concurrently.
Based on the logs from Azure Pipeline run
[#20261025.1|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=80472&view=logs&si=ae4f8708-9994-57d3-c2d7-b892156e7812],
the following three test jobs were running concurrently on the same Alibaba CI
machine (AlibabaCI004), assuming that the agent names share the same host.
||Test job||Agent||Start time||Duration||
|{{test_cron_adaptive_scheduler python}}|agent01|6:01 AM|1h 22m 48s|
|{{test_cron_hadoop343 connect}}|agent04|6:02 AM|38m 35s|
|{{test_ci tests}}|agent05|6:02 AM|56m 30s|
I also examined another set of jobs from the #20261001.3 pipeline run:
||Job name||Agent||Start time||End time||
|{{compile_cron_jdk21}}|agent03|5:45 AM|7:02:30 AM|
|{{test_ci python}}|agent05|7:11 AM|8:30:28 AM|
|{{test_cron_jdk21 python}}|agent01|7:48 AM|9:04:55 AM|
|{{test_cron_jdk21 table}}|agent03|7:50 AM|8:41:45 AM|
|{{test_ci core}}|agent02|8:17 AM|8:52:14 AM|
|{{test_ci table}}|agent04|8:30 AM|9:25:12 AM|
|{{test_ci tests}}|agent05|8:31 AM|9:31:04 AM|
*Peak overlap: 5 jobs*
Between 8:31 AM and 8:41:45 AM, all five agents were executing test jobs
concurrently.
Based on these observations, I would like to verify the actual VM allocation
and available resources, particularly how many agents can run concurrently on a
single VM and whether resource contention could be contributing to these
intermittent failures.
I also found that had been raised previously in [FLIP-396: Trial to test GitHub
Actions as an alternative for Flink's current Azure CI
infrastructure|https://cwiki.apache.org/confluence/spaces/FLINK/pages/278466632/FLIP-396+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink+s+current+Azure+CI+infrastructure]
(December 2023).
This makes me think that resource contention on the Azure CI infrastructure is
worth investigating further.
> RescaleTimelineITCase.testRecordRescaleForNewAvailableResource records three
> rescales instead of two
> ----------------------------------------------------------------------------------------------------
>
> Key: FLINK-40789
> URL: https://issues.apache.org/jira/browse/FLINK-40789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.3.1, 2.4.0, 2.5.0
> Reporter: Martijn Visser
> Assignee: Naveen Joseph
> Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79368&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1
> (master push 2026-09-23, test_ci core, JDK 17)
> {code}
> [ERROR]
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordRescaleForNewAvailableResource
> -- Time elapsed: 0.722 s <<< FAILURE!
> java.lang.AssertionError:
> Expected size: 2 but was: 3 in:
> [Rescale{... rescaleAttemptId=3, ... preRescaleParallelism=4,
> postRescaleParallelism=6 ... triggerCause=NEW_RESOURCE_AVAILABLE,
> terminalState=COMPLETED ...},
> Rescale{... rescaleAttemptId=2, ... preRescaleParallelism=2,
> postRescaleParallelism=4 ... triggerCause=NEW_RESOURCE_AVAILABLE,
> terminalState=COMPLETED ...},
> Rescale{... rescaleAttemptId=1, ... preRescaleParallelism=null,
> postRescaleParallelism=2 ... triggerCause=INITIAL_SCHEDULE,
> terminalState=COMPLETED ...}]
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.lambda$testRecordRescaleForNewAvailableResource$2(RescaleTimelineITCase.java:170)
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.runAdaptedParameterizedAssertion(RescaleTimelineITCase.java:397)
> at
> org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordRescaleForNewAvailableResource(RescaleTimelineITCase.java:164)
> {code}
>
>
> The test expects the initial schedule at parallelism 4 and
> one upscale to 6 after the third TaskManager starts. Here the job was first
> scheduled at 2 and went 2 -> 4 -> 6. The other RescaleTimelineITCase methods
> had similar races fixed in FLINK-40010, FLINK-40067 and FLINK-40076; this
> method has no ticket.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
